Oct 30 17:09:43.283218 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:09:43.283228 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:09:43.283237 ip-10-0-143-147 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:09:43.283574 ip-10-0-143-147 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:09:53.284833 ip-10-0-143-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:09:53.284854 ip-10-0-143-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0da6a39f24f04dbc8ec9feaa56bf8c3d -- Oct 30 17:11:56.898698 ip-10-0-143-147 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:11:57.333297 ip-10-0-143-147 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:57.333297 ip-10-0-143-147 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:11:57.333297 ip-10-0-143-147 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:57.333297 ip-10-0-143-147 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:11:57.339424 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339319 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.339424 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339422 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.339424 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339425 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.339424 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339429 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.339424 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339431 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339434 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339437 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339440 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339443 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339445 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339448 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339451 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339453 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339456 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339459 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339461 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339464 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339466 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339475 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339478 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339481 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339483 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339486 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339488 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.339566 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339491 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339493 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339496 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339499 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339504 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339508 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339511 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339514 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339517 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339519 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339522 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339524 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339529 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339532 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339536 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339539 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339542 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339545 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339547 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.340118 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339550 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339553 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339555 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339558 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339561 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339563 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339566 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339568 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339571 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339574 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339577 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339579 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339582 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339585 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339587 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339590 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339593 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339595 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339598 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.340812 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339600 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339603 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339605 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339608 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339611 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339614 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339617 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339620 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339624 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339626 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339629 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339633 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339636 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339639 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339641 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339644 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339646 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339649 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339652 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.341265 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339654 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339657 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339659 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339663 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339666 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339668 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339671 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339673 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339676 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339678 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339681 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339684 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339686 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339688 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339691 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339693 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339696 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339698 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339701 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.341724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339703 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.342247 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339706 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.342247 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339709 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.342247 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339712 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.342247 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339716 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.342247 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.339719 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.342555 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342537 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342556 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342565 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342568 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342571 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342576 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342581 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342584 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342587 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342590 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.342592 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342593 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342598 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342601 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342604 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342606 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342612 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342615 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342618 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342620 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342623 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342625 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342628 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342631 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342633 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342636 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342638 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342641 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342644 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342649 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.342867 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342651 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342654 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342657 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342659 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342662 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342664 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342667 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342669 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342672 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342674 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342677 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342679 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342684 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342687 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342691 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342695 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342697 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342700 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342703 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342706 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.343311 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342709 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342711 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342714 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342717 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342722 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342724 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342727 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342729 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342732 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342735 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342737 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342740 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342742 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342745 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342747 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342750 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342753 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342758 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342761 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342763 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.343796 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342766 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342768 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342774 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342777 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342780 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342784 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342788 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342790 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342794 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342799 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342802 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342805 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342808 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342810 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342813 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342816 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342819 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342821 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.344302 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342824 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342828 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342833 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342836 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342838 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342841 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342844 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342847 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342850 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342852 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342855 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342857 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342860 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342863 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342865 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342870 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342873 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342876 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.342879 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343690 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.344735 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343695 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343698 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343702 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343706 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343709 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343714 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343717 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343720 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343722 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343725 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343728 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343730 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343733 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343736 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343738 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343742 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343744 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343747 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343752 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343755 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.345204 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343758 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343760 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343763 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343765 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343768 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343771 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343773 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343776 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343778 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343781 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343784 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343789 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343792 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343795 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343798 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343801 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343804 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343807 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343810 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343812 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.345724 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343815 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343818 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343820 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343823 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343828 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343831 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343835 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343839 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343842 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343844 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343847 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343850 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343852 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343855 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343857 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343860 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343865 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343868 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343870 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.346186 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343873 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343875 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343878 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343881 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343884 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343887 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343890 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343892 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343895 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343898 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343903 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343906 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343909 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343911 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343914 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343918 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343920 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343923 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343926 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343928 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.346647 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343931 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343933 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343939 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343941 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343943 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343946 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343949 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343951 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343954 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343956 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343959 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343961 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343964 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343967 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343969 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343975 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343978 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343981 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343984 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343987 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.347109 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343990 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343992 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343996 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.343999 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.344002 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.344007 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345206 2563 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345362 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345380 2563 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345385 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345391 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345394 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345399 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345404 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345407 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345411 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345414 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345418 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345421 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345424 2563 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345427 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345430 2563 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345433 2563 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:11:57.347587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345436 2563 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345442 2563 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345445 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345449 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345452 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345456 2563 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345460 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345463 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345466 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345470 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345473 2563 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345476 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345480 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345484 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345487 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345493 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345497 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345500 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345503 2563 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345507 2563 flags.go:64] FLAG: --enable-server="true" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345510 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345517 2563 flags.go:64] FLAG: --event-burst="100" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345520 2563 flags.go:64] FLAG: --event-qps="50" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345525 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:11:57.348142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345530 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345533 2563 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345540 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345543 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345546 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345550 2563 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345553 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345556 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345559 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345562 2563 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345565 2563 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345568 2563 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345571 2563 flags.go:64] FLAG: --feature-gates="" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345575 2563 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345578 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345582 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345585 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345588 2563 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345592 2563 flags.go:64] FLAG: --help="false" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345595 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345598 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345602 2563 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345605 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345610 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:11:57.348727 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345614 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345617 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345620 2563 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345623 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345626 2563 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345629 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345632 2563 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345635 2563 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345638 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345641 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345645 2563 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345648 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345651 2563 flags.go:64] FLAG: --lock-file="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345654 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345657 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345660 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345666 2563 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345669 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345672 2563 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345675 2563 flags.go:64] FLAG: --logging-format="text" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345678 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345682 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345685 2563 flags.go:64] FLAG: --manifest-url="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345688 2563 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345695 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:11:57.349331 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345699 2563 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345702 2563 flags.go:64] FLAG: --max-pods="110" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345705 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345708 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345712 2563 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345715 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345719 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345722 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345725 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345734 2563 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345738 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345741 2563 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345744 2563 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345747 2563 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345750 2563 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345753 2563 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345756 2563 flags.go:64] FLAG: --port="10250" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345759 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345762 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd063e6ef3b21223" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345765 2563 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345768 2563 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345771 2563 flags.go:64] FLAG: --register-node="true" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345774 2563 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345778 2563 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345781 2563 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345785 2563 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:11:57.349927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345788 2563 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345791 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345795 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345798 2563 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345801 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345805 2563 flags.go:64] FLAG: --runonce="false" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345808 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345811 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345814 2563 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345817 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345820 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345823 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345826 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345830 2563 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345832 2563 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345836 2563 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345838 2563 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345841 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345844 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345847 2563 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345850 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345857 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345860 2563 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345863 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345867 2563 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:11:57.350549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345870 2563 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345873 2563 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345877 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345880 2563 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345883 2563 flags.go:64] FLAG: --v="2" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345888 2563 flags.go:64] FLAG: --version="false" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345892 2563 flags.go:64] FLAG: --vmodule="" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345897 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.345901 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346082 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346088 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346092 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346096 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346099 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346102 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346105 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346107 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346110 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346113 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346116 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346118 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.351123 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346121 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346124 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346127 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346129 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346132 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346137 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346141 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346144 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346147 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346150 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346153 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346156 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346159 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346162 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346164 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346167 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346169 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346172 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346174 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.351660 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346180 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346183 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346186 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346189 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346192 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346194 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346198 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346200 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346203 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346206 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346209 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346211 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346214 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346216 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346219 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346221 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346224 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346227 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346229 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346233 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.352431 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346235 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346238 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346240 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346243 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346246 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346248 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346251 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346254 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346256 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346259 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346262 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346264 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346268 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346271 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346273 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346278 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346281 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346284 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346286 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346289 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.353294 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346291 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346294 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346297 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346300 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346302 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346305 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346307 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346310 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346313 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346315 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346318 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346321 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346324 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346326 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346329 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346332 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346334 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346337 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346339 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346342 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.354177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346345 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346347 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346350 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346352 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346357 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346359 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346362 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346366 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346380 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346383 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346386 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346388 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346391 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346394 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346396 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.355061 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.346428 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346470 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346475 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346478 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346480 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346483 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346486 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346488 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346491 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346494 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346498 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346500 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346503 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346506 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346508 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346511 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346514 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346516 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346519 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346521 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.355892 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346524 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346527 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346531 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346533 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346537 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346540 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346542 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346545 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346548 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346551 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346553 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346556 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346559 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346561 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346564 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346567 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346569 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346572 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346574 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346577 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.356742 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346579 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346582 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346585 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346588 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346591 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346594 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346597 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346599 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346602 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346604 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346607 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346610 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346612 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346615 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346619 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346621 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346625 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346628 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346630 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.357420 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346633 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346636 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346639 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346641 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346644 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346646 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346649 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346652 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346654 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346657 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346660 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346663 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346666 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346669 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346672 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346674 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346677 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346680 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346683 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.358190 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346685 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346688 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346691 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346693 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346696 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346698 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346702 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346706 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346710 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346712 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346716 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346719 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346722 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346724 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346727 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346729 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346732 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346734 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346737 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.358813 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346739 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346742 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346744 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346748 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346752 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346756 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346758 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346761 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346763 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.346767 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.354321 2563 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.354346 2563 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.354387 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.354395 2563 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354487 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354496 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354501 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354507 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.359309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354512 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354517 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354521 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354525 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354530 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354534 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354538 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354543 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354548 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354552 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354557 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354561 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354565 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354570 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354574 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354578 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354583 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354587 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354592 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354596 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.359769 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354601 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354606 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354610 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354615 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354619 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354625 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354629 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354634 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354638 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354642 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354648 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354652 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354656 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354660 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354665 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354670 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354674 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354678 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354683 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354687 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.360309 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354692 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354696 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354700 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354705 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354709 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354713 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354718 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354722 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354726 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354731 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354735 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354739 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354744 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354749 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354754 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354759 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354763 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354767 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354776 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.360998 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354781 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354785 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354789 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354794 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354798 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354802 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354807 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354811 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354815 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354821 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354829 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354833 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354838 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354842 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354846 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354851 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354855 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354859 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354864 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.361493 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354868 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354872 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354876 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354881 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354886 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354890 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354894 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354899 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354904 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354908 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354913 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354917 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354921 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354927 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354931 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354936 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354942 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354950 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354955 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.361982 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354959 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354964 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354968 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354972 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.354976 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.355023 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355260 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355268 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355273 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355277 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355281 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355286 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355290 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355295 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355299 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355304 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.362525 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355309 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355313 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355317 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355322 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355326 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355331 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355335 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355340 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355344 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355349 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355353 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355358 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355363 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355386 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355391 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355395 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355399 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355403 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355407 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.362899 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355410 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355415 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355419 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355423 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355427 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355431 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355434 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355438 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355442 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355446 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355450 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355454 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355458 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355463 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355467 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355471 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355475 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355479 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355483 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355487 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.363676 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355491 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355495 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355499 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355503 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355509 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355515 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355520 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355525 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355529 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355533 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355537 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355541 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355545 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355550 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355554 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355558 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355564 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355569 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355573 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355577 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.364586 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355581 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355586 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355591 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355596 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355600 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355605 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355609 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355613 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355619 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355623 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355628 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355632 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355636 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355641 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355645 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355649 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355654 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355658 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355663 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355668 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.365430 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355672 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355677 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355681 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355686 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355690 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355697 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355704 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355708 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355713 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355718 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355722 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355727 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355731 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355735 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355740 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355745 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.366199 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355749 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.355798 2563 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355854 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355862 2563 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355867 2563 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355871 2563 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355875 2563 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355880 2563 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355884 2563 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355889 2563 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355893 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355897 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355901 2563 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355906 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355911 2563 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355916 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355920 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:11:57.366658 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355924 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355929 2563 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355934 2563 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355938 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355942 2563 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355946 2563 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355951 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355955 2563 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355959 2563 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355963 2563 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355968 2563 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355973 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355977 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355982 2563 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355986 2563 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355990 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355995 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.355999 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356004 2563 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356008 2563 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:11:57.367189 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356013 2563 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356017 2563 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356021 2563 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356026 2563 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356030 2563 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356036 2563 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356042 2563 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356047 2563 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356051 2563 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356057 2563 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356062 2563 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356068 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356072 2563 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356077 2563 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356081 2563 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356085 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356090 2563 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356094 2563 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356099 2563 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:11:57.367727 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356103 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356108 2563 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356115 2563 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356119 2563 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356123 2563 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356128 2563 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356132 2563 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356136 2563 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356141 2563 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356146 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356151 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356155 2563 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356159 2563 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356163 2563 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356168 2563 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356172 2563 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356176 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356180 2563 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356185 2563 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:11:57.368220 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356189 2563 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356193 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356198 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356202 2563 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356207 2563 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356210 2563 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356215 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356220 2563 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356224 2563 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356228 2563 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356233 2563 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356237 2563 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356242 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356246 2563 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356250 2563 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356256 2563 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356260 2563 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356264 2563 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356269 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:11:57.368683 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356273 2563 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356278 2563 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356282 2563 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356286 2563 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356291 2563 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356296 2563 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356300 2563 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356304 2563 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356308 2563 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356312 2563 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356317 2563 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356321 2563 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356325 2563 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:57.356333 2563 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.356478 2563 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.360470 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.361472 2563 server.go:1014] "Starting client certificate rotation" Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.361568 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:11:57.369131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.361609 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:11:57.369613 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.363696 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:11:57.369613 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.366086 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:11:57.375409 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.375391 2563 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:11:57.378742 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.378719 2563 log.go:25] "Validated CRI v1 image API" Oct 30 17:11:57.379523 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.379499 2563 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:11:57.382708 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.382690 2563 fs.go:135] Filesystem UUIDs: map[3896f177-1054-4d15-8f42-2651d1c0aa7a:/dev/nvme0n1p4 6dbb4951-afda-417f-9324-56e1a56584b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:11:57.382768 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.382708 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:11:57.387066 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.386954 2563 manager.go:217] Machine: {Timestamp:2026-10-30 17:11:57.386477133 +0000 UTC m=+0.349936363 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096097 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279fffb43a539a578993507e1c6932 SystemUUID:ec279fff-b43a-539a-5789-93507e1c6932 BootID:0da6a39f-24f0-4dbc-8ec9-feaa56bf8c3d Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:8d:a1:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:8d:a1:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:2b:b2:af:f3:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:11:57.387066 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.387060 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:11:57.387168 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.387141 2563 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:11:57.390554 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.390523 2563 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:11:57.390693 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.390555 2563 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:11:57.391161 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.391152 2563 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:11:57.391198 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.391163 2563 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:11:57.391198 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.391177 2563 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:11:57.391198 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.391190 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:11:57.391900 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.391890 2563 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:11:57.394469 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.394456 2563 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:11:57.394591 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.394582 2563 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:11:57.395230 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395220 2563 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:11:57.395277 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395234 2563 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:11:57.395277 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395249 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:11:57.395277 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395258 2563 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:11:57.395277 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395266 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:11:57.395650 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.395626 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:11:57.397531 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.397516 2563 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:11:57.506781 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.506735 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:11:57.506948 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.506843 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:11:57.538241 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.538211 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:11:57.538956 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.538941 2563 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:11:57.538997 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.538973 2563 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:11:57.539186 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539175 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:11:57.539237 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539191 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:11:57.539237 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539197 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:11:57.539237 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539203 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:11:57.539237 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539208 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:11:57.539237 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539213 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:11:57.539641 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539622 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:11:57.539641 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539641 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:11:57.539641 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539650 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:11:57.539757 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539656 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:11:57.539757 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539666 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:11:57.539757 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.539674 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:11:57.540750 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.540741 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:11:57.540750 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.540750 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:11:57.544674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.544658 2563 server.go:1263] "Started kubelet" Oct 30 17:11:57.544815 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.544697 2563 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:11:57.544862 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.544794 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:11:57.544898 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.544882 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:11:57.545312 ip-10-0-143-147 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:11:57.547597 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.547575 2563 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:11:57.551827 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.551812 2563 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:11:57.554924 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.554900 2563 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:11:57.555568 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.555549 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:11:57.556447 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.556430 2563 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:11:57.556588 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.556565 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:11:57.556950 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.556923 2563 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:11:57.556950 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.556949 2563 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:11:57.557100 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.556986 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:11:57.557100 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.557038 2563 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:11:57.557100 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.557049 2563 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:11:57.558054 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558039 2563 factory.go:55] Registering systemd factory Oct 30 17:11:57.558131 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558057 2563 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:11:57.558295 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558282 2563 factory.go:153] Registering CRI-O factory Oct 30 17:11:57.558336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558299 2563 factory.go:223] Registration of the crio container factory successfully Oct 30 17:11:57.558385 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558354 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:11:57.558385 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558384 2563 factory.go:103] Registering Raw factory Oct 30 17:11:57.558518 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.558399 2563 manager.go:1199] Started watching for new ooms in manager Oct 30 17:11:57.559244 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.559231 2563 manager.go:319] Starting recovery of all containers Oct 30 17:11:57.568845 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.568824 2563 manager.go:324] Recovery completed Oct 30 17:11:57.573453 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.573436 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.575728 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.575690 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.575821 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.575744 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.575821 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.575760 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.576818 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.576801 2563 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:11:57.576818 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.576815 2563 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:11:57.576966 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.576822 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.576966 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.576838 2563 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:11:57.577668 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.577653 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.577787 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.577682 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.577787 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.577696 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.578363 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.578351 2563 policy_none.go:50] "Start" Oct 30 17:11:57.578418 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.578382 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:11:57.578418 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.578395 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:11:57.579428 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.579417 2563 policy_none.go:44] "Start" Oct 30 17:11:57.588887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.588869 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:11:57.589038 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.589020 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:11:57.589844 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.588842 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a66a1c7b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.544455291 +0000 UTC m=+0.507914520,LastTimestamp:2026-10-30 17:11:57.544455291 +0000 UTC m=+0.507914520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.611512 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.611490 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:11:57.611613 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.611493 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.614952 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.614926 2563 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:11:57.618743 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.618727 2563 manager.go:341] "Starting Device Plugin manager" Oct 30 17:11:57.618841 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.618761 2563 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:11:57.618841 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.618771 2563 server.go:87] "Starting device plugin registration server" Oct 30 17:11:57.619053 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.619039 2563 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:11:57.619100 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.619056 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:11:57.619193 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.619173 2563 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:11:57.619289 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.619263 2563 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:11:57.619289 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.619277 2563 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:11:57.619827 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.619808 2563 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:11:57.619918 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.619850 2563 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:11:57.639853 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.639766 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.666871 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.658633 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.692415 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.692306 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.577668378 +0000 UTC m=+0.541127608,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.713466 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.713355 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.577688192 +0000 UTC m=+0.541147425,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.720085 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.720065 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.720980 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.720962 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.721087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.720999 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.721087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.721016 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.721087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.721052 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.737428 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.737252 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:11:57.738635 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.738618 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:11:57.738726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.738640 2563 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:11:57.738726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.738662 2563 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:11:57.738726 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.738701 2563 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:11:57.745020 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.744924 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.577701683 +0000 UTC m=+0.541160918,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.759756 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.759726 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.759899 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.759725 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5aab9a6da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.616776922 +0000 UTC m=+0.580236143,LastTimestamp:2026-10-30 17:11:57.616776922 +0000 UTC m=+0.580236143,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.793764 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.793708 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:11:57.793764 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.793670 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5aad70662 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.618701922 +0000 UTC m=+0.582161145,LastTimestamp:2026-10-30 17:11:57.618701922 +0000 UTC m=+0.582161145,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.809874 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.809846 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Oct 30 17:11:57.809962 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.809853 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.720979511 +0000 UTC m=+0.684438742,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.839108 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.839043 2563 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"] Oct 30 17:11:57.839179 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.839116 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.840029 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.840013 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.840091 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.840042 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.840091 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.840053 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.841666 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.841653 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.841838 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.841819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.841939 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.841850 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.842396 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842366 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.842480 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842413 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.842480 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842415 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.842480 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842435 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.842480 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842441 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.842480 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.842447 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.843600 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.843586 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.843663 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.843610 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.844094 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.844023 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.721006949 +0000 UTC m=+0.684466179,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.844340 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.844322 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.844437 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.844356 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.844437 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.844393 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.860224 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.860146 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.721021614 +0000 UTC m=+0.684480847,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.879703 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.879675 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.883235 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.883214 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.893463 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.893383 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.840029202 +0000 UTC m=+0.803488431,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.927689 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.927601 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.840046581 +0000 UTC m=+0.803505810,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.943509 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.943428 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.840056821 +0000 UTC m=+0.803516051,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.958357 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.958328 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.958442 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.958362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.958442 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.958391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.960422 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.960405 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:57.961505 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.961486 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:57.961561 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.961523 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:57.961561 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.961538 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:57.961629 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:57.961571 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.975847 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.975767 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.842395964 +0000 UTC m=+0.805855203,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:57.999606 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.999568 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:57.999765 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:57.999682 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.842419014 +0000 UTC m=+0.805878249,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:58.014264 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:58.014181 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.842426585 +0000 UTC m=+0.805885814,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.251413 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.251327 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.842440672 +0000 UTC m=+0.805899903,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.258308 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.258432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.258432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.258432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258369 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.258432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258383 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.258432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.258422 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.294991 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.294856 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.842448869 +0000 UTC m=+0.805908103,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.337923 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.337838 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.842451231 +0000 UTC m=+0.805910460,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.382424 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.382391 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.385866 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.385500 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.392377 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.392298 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.844339092 +0000 UTC m=+0.807798326,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.445336 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.445302 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Oct 30 17:11:59.445336 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.445242 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847a6a3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847a6a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575751331 +0000 UTC m=+0.539210563,LastTimestamp:2026-10-30 17:11:57.844363382 +0000 UTC m=+0.807822614,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.464806 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.464726 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a847e26a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a847e26a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575766634 +0000 UTC m=+0.539225866,LastTimestamp:2026-10-30 17:11:57.844398391 +0000 UTC m=+0.807857620,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.497879 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.497794 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-147.ec2.internal.18e35de5a8474f42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-147.ec2.internal.18e35de5a8474f42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-147.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:57.575728962 +0000 UTC m=+0.539188197,LastTimestamp:2026-10-30 17:11:57.961506512 +0000 UTC m=+0.924965745,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.598816 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.598731 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:11:59.599774 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.599756 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:11:59.599887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.599785 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:11:59.599887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.599795 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:11:59.599887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.599824 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.662334 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.662300 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:11:59.836751 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.836721 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:11:59.893756 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:59.893720 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c50f6fe0fd9f68f1ab538d873faa96.slice/crio-faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 WatchSource:0}: Error finding container faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521: Status 404 returned error can't find the container with id faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 Oct 30 17:11:59.897976 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.897959 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:11:59.929155 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.929072 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal.18e35de632c0d7fe openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal,UID:01c50f6fe0fd9f68f1ab538d873faa96,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:59.89894963 +0000 UTC m=+1.663518419,LastTimestamp:2026-10-30 17:11:59.89894963 +0000 UTC m=+1.663518419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:11:59.932639 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:11:59.932604 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33e219052cd1905654f6e2cbcaae521.slice/crio-f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488 WatchSource:0}: Error finding container f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488: Status 404 returned error can't find the container with id f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488 Oct 30 17:11:59.941197 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.941152 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"a33e219052cd1905654f6e2cbcaae521","Type":"ContainerStarted","Data":"f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488"} Oct 30 17:11:59.942273 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:11:59.942240 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521"} Oct 30 17:11:59.948732 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.948709 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:11:59.948828 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:11:59.948736 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-147.ec2.internal.18e35de634efccf2 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-147.ec2.internal,UID:a33e219052cd1905654f6e2cbcaae521,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:11:59.935581426 +0000 UTC m=+1.700150214,LastTimestamp:2026-10-30 17:11:59.935581426 +0000 UTC m=+1.700150214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:00.006434 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.006394 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:00.007751 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.007732 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:00.007864 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.007762 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:00.007864 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.007771 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:00.163774 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:00.163690 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:12:00.305926 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:00.305891 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Oct 30 17:12:00.357672 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:00.357633 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:12:00.404841 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:00.404801 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:12:00.463147 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.463058 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:00.467087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.467060 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:00.467221 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.467099 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:00.467221 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.467113 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:00.467221 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.467148 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:00.561738 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:00.561705 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:00.835704 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:00.835621 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:01.163718 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.163627 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal.18e35de67b09abc3 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal,UID:01c50f6fe0fd9f68f1ab538d873faa96,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.212s (1.212s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.111681987 +0000 UTC m=+2.876250778,LastTimestamp:2026-10-30 17:12:01.111681987 +0000 UTC m=+2.876250778,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.398127 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.398036 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-147.ec2.internal.18e35de688be2f7b kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-147.ec2.internal,UID:a33e219052cd1905654f6e2cbcaae521,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.406s (1.406s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.341615995 +0000 UTC m=+3.106184788,LastTimestamp:2026-10-30 17:12:01.341615995 +0000 UTC m=+3.106184788,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.444393 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.444297 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal.18e35de688be6499 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal,UID:01c50f6fe0fd9f68f1ab538d873faa96,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.341629593 +0000 UTC m=+3.106198401,LastTimestamp:2026-10-30 17:12:01.341629593 +0000 UTC m=+3.106198401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.491441 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.491342 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal.18e35de6893802b1 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal,UID:01c50f6fe0fd9f68f1ab538d873faa96,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.349599921 +0000 UTC m=+3.114168710,LastTimestamp:2026-10-30 17:12:01.349599921 +0000 UTC m=+3.114168710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.537749 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.537658 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-147.ec2.internal.18e35de68ae02754 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-147.ec2.internal,UID:a33e219052cd1905654f6e2cbcaae521,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.377396564 +0000 UTC m=+3.141965369,LastTimestamp:2026-10-30 17:12:01.377396564 +0000 UTC m=+3.141965369,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.587734 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.587646 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-143-147.ec2.internal.18e35de68b3616f1 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-143-147.ec2.internal,UID:a33e219052cd1905654f6e2cbcaae521,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-143-147.ec2.internal,},FirstTimestamp:2026-10-30 17:12:01.383028465 +0000 UTC m=+3.147597277,LastTimestamp:2026-10-30 17:12:01.383028465 +0000 UTC m=+3.147597277,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-147.ec2.internal,}" Oct 30 17:12:01.834760 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.834689 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:01.947279 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.947239 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655"} Oct 30 17:12:01.948165 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.947288 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:01.948253 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.948191 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:01.948253 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.948217 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:01.948253 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.948231 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:01.948422 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.948410 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:01.948762 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.948742 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"a33e219052cd1905654f6e2cbcaae521","Type":"ContainerStarted","Data":"ecc98d87375ffdda340fa2e6b6f44b2acca0c1736bc658148eb316e062735c09"} Oct 30 17:12:01.948907 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.948792 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:01.949570 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.949556 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:01.949653 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.949583 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:01.949653 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:01.949597 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:01.949768 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.949754 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:01.951109 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:01.951092 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Oct 30 17:12:02.162448 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.162378 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:02.163507 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.163491 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:02.163574 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.163524 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:02.163574 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.163534 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:02.163574 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.163560 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:02.263447 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:02.263411 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:02.263447 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:02.263427 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:12:02.338781 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.338749 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:02.339848 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.339827 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:02.339962 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.339864 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:02.339962 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.339875 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:02.845172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.845144 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:02.892001 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:02.891967 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:12:02.950238 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.950211 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:02.950645 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.950300 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:02.952337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952318 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:02.952446 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952350 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:02.952446 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952320 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:02.952446 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952359 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:02.952446 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952379 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:02.952446 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:02.952391 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:02.952593 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:02.952564 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:02.952668 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:02.952656 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:03.199694 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:03.199663 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:12:03.495920 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:03.495833 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:12:03.842143 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:03.842117 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:04.840090 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:04.840050 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:05.463908 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.463875 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:05.467321 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.467294 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:05.467433 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.467329 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:05.467433 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.467341 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:05.467433 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.467373 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:05.517063 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:05.517032 2563 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Oct 30 17:12:05.565614 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:05.565583 2563 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:05.842732 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:05.842703 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:06.847763 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:06.847728 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:06.895851 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:06.895822 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 17:12:07.164398 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:07.164305 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:07.165352 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:07.165333 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:07.165470 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:07.165362 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:07.165470 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:07.165374 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:07.834273 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:07.834225 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:12:07.834421 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:07.834278 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:12:08.572187 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:08.572147 2563 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:12:08.648225 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.648188 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:12:08.648784 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.648764 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nzwm4" Oct 30 17:12:08.660817 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.660795 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nzwm4" Oct 30 17:12:08.819098 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:08.819061 2563 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:08.852725 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.852654 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:08.878398 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.878374 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:08.946591 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:08.946562 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.216783 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.216706 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.216783 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:09.216741 2563 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.263395 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.263367 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.288837 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.288817 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.355389 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.355366 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.503958 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.503882 2563 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:09.560067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.560029 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:12:09.560227 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.560206 2563 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:12:09.599517 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.599489 2563 apiserver.go:52] "Watching apiserver" Oct 30 17:12:09.643202 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.643173 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.643202 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:09.643198 2563 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.644440 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.644426 2563 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:12:09.644486 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.644467 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=[] Oct 30 17:12:09.656066 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.656038 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:12:09.663749 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.663718 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:07:08 +0000 UTC" deadline="2028-05-04 05:09:31.210567948 +0000 UTC" Oct 30 17:12:09.663749 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.663747 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13235h57m21.546825044s" Oct 30 17:12:09.918432 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.918400 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:09.942002 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:09.941977 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:10.005623 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:10.005600 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:10.279633 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:10.279558 2563 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:10.279633 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:10.279583 2563 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-143-147.ec2.internal" not found Oct 30 17:12:11.942787 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:11.942750 2563 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:11.965970 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.965947 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:11.966777 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.966763 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:11.966851 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.966793 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:11.966851 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.966802 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:11.966851 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.966832 2563 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:11.983445 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:11.983428 2563 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:11.983486 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:11.983450 2563 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-147.ec2.internal\": node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.024139 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.024116 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.025634 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.025622 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gb9j4"] Oct 30 17:12:12.025779 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.025770 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.026633 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.026617 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.026678 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.026647 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.026678 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.026657 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.028910 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.028896 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-wlnpk"] Oct 30 17:12:12.029003 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.028992 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.029036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.029075 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029057 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.029730 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029714 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.029730 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029728 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.029873 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029747 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.029873 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029757 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.029873 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029744 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.029873 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.029836 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.031019 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.030998 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:12.032333 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.031336 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.033543 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.032608 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.033543 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.032655 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.033543 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.032686 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.033543 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.032768 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:12.044011 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.043979 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:12:12.044011 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.043992 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.044011 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.044000 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.044011 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.043989 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:12:12.044186 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.043985 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:12:12.044221 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.044191 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\"" Oct 30 17:12:12.054532 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.054509 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:12:12.113741 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.113712 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lqgx2"] Oct 30 17:12:12.113907 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.113894 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.114853 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.114839 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.114933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.114869 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.114933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.114879 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.116113 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116100 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.116148 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116134 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.116756 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116734 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-m72vs"] Oct 30 17:12:12.116869 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116859 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.116969 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116954 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.117010 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116979 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.117010 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.116990 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.117076 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.117054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:12.117584 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.117569 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.117665 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.117597 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.117665 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.117610 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.118918 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.118902 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.119000 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.118939 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.119705 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.119692 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.119779 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.119720 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.119779 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.119741 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.124337 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.124321 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.129411 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129489 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129489 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129489 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129447 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129489 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:12.129607 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129607 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129539 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129607 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129607 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcc2d\" (UniqueName: \"kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129717 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129628 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129717 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129645 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129717 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129717 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129701 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129825 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129825 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129825 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129825 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129855 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.129933 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.129904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.138807 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.138783 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lr6fs"] Oct 30 17:12:12.139041 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.139027 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.139770 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.139751 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.139868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.139787 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.139868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.139800 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.141035 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.141021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.141081 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.141048 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.141824 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.141809 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.141903 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.141835 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.141903 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.141844 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.144272 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.144235 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.144352 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.144305 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.144352 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.144336 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\"" Oct 30 17:12:12.216523 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.216429 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rzvbg"] Oct 30 17:12:12.216741 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.216601 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.217619 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.217598 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.217713 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.217636 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.217713 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.217650 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.219417 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.219392 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.219509 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.219428 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.219509 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.219442 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2k8mk"] Oct 30 17:12:12.219603 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.219592 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.220232 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220214 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.220317 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220241 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.220317 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220255 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.220317 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220221 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.220452 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220317 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.220452 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.220336 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.221971 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.221954 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.221971 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.221980 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.222802 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.222788 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.222869 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.222814 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.222869 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.222825 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.224870 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.224854 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.230496 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.230462 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.230761 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.230383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.230920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.230961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.230993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcc2d\" (UniqueName: \"kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231067 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231132 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231159 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.231288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231251 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7qb\" (UniqueName: \"kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231351 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231386 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw7s\" (UniqueName: \"kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.231674 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231852 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgdl\" (UniqueName: \"kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231887 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231941 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.231982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232024 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232072 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232250 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232366 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.232995 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.232451 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233171 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233149 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233332 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233194 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233395 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233329 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233395 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233381 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.233497 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233381 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233497 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233424 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233497 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233455 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233665 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233501 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233716 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233694 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.233939 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.233908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.234058 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234046 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.234093 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234072 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234464 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234606 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234741 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234742 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234760 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234776 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234807 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.234977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.234956 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.235338 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.235317 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.235338 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.235319 2563 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:12:12.235412 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.235341 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.235412 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.235356 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.239309 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.239158 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.244546 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244526 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:12:12.244661 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244562 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.244661 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244579 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.244661 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244617 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244706 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244710 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244737 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244744 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244761 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244764 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244774 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\"" Oct 30 17:12:12.244835 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.244805 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\"" Oct 30 17:12:12.254170 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.254154 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:12.254227 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.254176 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:12.254227 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.254188 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:12.254303 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.254247 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.754232332 +0000 UTC m=+14.518801134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:12.256626 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.256598 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcc2d\" (UniqueName: \"kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.313336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.313301 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-w4gnb"] Oct 30 17:12:12.313489 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.313476 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.315525 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.315511 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.315587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.315538 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.315587 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.315551 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.317752 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.317738 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl"] Oct 30 17:12:12.317855 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.317839 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.317903 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.317869 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.317951 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.317937 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.318762 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318743 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.318854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318774 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.318854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318784 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.318854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318750 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.318854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318804 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.318854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.318813 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.320562 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.320548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.320620 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.320574 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.321277 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.321250 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.321340 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.321292 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.321340 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.321309 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.325024 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.325009 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.332307 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.332307 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332339 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332387 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332430 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7h6\" (UniqueName: \"kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7qb\" (UniqueName: \"kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332546 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw7s\" (UniqueName: \"kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332589 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332631 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332656 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332679 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.332707 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332774 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.332978 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgdl\" (UniqueName: \"kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333086 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333160 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333225 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333320 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333358 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333591 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333605 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.333642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333627 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333614 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333740 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333769 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333778 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333784 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333807 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333822 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333869 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.333936 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.334021 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:12.334067 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.334072 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:12.834054685 +0000 UTC m=+14.598623504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:12.334427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.334126 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.334427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.334145 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.334427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.334153 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.335790 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.335776 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-77wmf"] Oct 30 17:12:12.335923 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.335914 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.336581 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.336567 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.336628 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.336590 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.336628 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.336599 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.337783 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.337771 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.337826 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.337794 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.338493 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.338480 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.338567 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.338501 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.338567 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.338510 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.345002 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.344983 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.345002 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.344992 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.344983 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.345041 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.345057 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.345059 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.345081 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\"" Oct 30 17:12:12.345157 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.345121 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:12:12.348317 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.348303 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:12.356089 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.356071 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgdl\" (UniqueName: \"kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.356186 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.356138 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7qb\" (UniqueName: \"kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.356339 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.356321 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw7s\" (UniqueName: \"kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.356915 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.356896 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee63db4_03fa_415b_9489_b188e6a71575.slice/crio-c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7 WatchSource:0}: Error finding container c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7: Status 404 returned error can't find the container with id c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7 Oct 30 17:12:12.360152 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.360137 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-854ml"] Oct 30 17:12:12.360370 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.360351 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.361147 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.361129 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.361217 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.361164 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.361217 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.361180 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.363892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.363876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.363947 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.363912 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.364670 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.364653 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.364735 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.364687 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.364735 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.364702 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.364793 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.364781 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:12.411574 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.411551 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bgdc6"] Oct 30 17:12:12.411791 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.411779 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.412848 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.412828 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.412918 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.412864 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.412918 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.412877 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.414461 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.414446 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.414507 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.414480 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.415343 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.415327 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.415403 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.415358 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.415403 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.415372 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.425417 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.425385 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.427592 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.427570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:12.433527 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433504 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.433627 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433627 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433560 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7h6\" (UniqueName: \"kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433719 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433668 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.433719 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433701 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.433793 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.433793 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433793 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433793 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433784 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433805 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433843 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433853 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.433964 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.433961 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434021 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434051 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434118 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434191 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434168 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434411 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434482 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434502 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434583 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434631 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434779 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpsx\" (UniqueName: \"kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434839 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434867 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.434887 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434879 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434949 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.434984 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.435008 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.435023 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.435427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.435004 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.436365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.435828 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.436365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.435975 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.436365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436007 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.436365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436024 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.436594 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.436643 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.436688 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.436733 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.436733 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.436807 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57bx\" (UniqueName: \"kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.436807 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.436892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436810 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.436892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.436892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436877 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.437005 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436932 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.437005 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436979 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.437005 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.436984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.437113 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.437113 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnkr\" (UniqueName: \"kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.437185 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.437185 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437163 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.437364 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.437524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437500 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.437615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437600 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.437866 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.437845 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.438908 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.438889 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.438998 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.438925 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.438998 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.438941 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.439293 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.439247 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.440568 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.440504 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.441190 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.441169 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.444852 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.444829 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:12:12.444938 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.444906 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:12:12.444938 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.444924 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:12:12.445012 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.444940 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:12:12.445042 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.445020 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\"" Oct 30 17:12:12.445042 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.445036 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\"" Oct 30 17:12:12.455192 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.455175 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:12.459521 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.459503 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7h6\" (UniqueName: \"kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.464177 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.464150 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26243dda_d8a0_457d_8844_81a32a6def00.slice/crio-245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c WatchSource:0}: Error finding container 245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c: Status 404 returned error can't find the container with id 245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c Oct 30 17:12:12.472029 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.472007 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc52026_7904_40fd_95ec_1c7b3db1b5cf.slice/crio-9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827 WatchSource:0}: Error finding container 9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827: Status 404 returned error can't find the container with id 9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827 Oct 30 17:12:12.525836 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.525803 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.530031 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.530007 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:12.536774 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.536758 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:12.537064 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.537039 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadcc9ba_2535_48cd_abef_1535e15f6ec6.slice/crio-a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b WatchSource:0}: Error finding container a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b: Status 404 returned error can't find the container with id a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b Oct 30 17:12:12.537717 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537766 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537780 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537790 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.537811 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537801 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57bx\" (UniqueName: \"kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdv9\" (UniqueName: \"kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537912 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537934 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538072 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnkr\" (UniqueName: \"kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.537908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538190 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538228 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538290 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538341 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538349 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538362 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538417 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.538427 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538431 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538470 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538480 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538473 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538521 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538576 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538589 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538609 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538660 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538689 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538741 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539336 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538806 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpsx\" (UniqueName: \"kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538837 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538859 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538782 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538929 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538957 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.538971 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.539064 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.539088 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.539868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.539346 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.540328 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540010 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.540328 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540036 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.540328 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540057 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.540456 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540441 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.540485 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540471 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.540512 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540486 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.540863 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540848 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.540907 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540875 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.540907 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.540890 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.540996 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.540984 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:12.541044 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.541034 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:13.041015777 +0000 UTC m=+14.805584586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:12.541114 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.541101 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.541148 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.541126 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.541148 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.541140 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.544686 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.544666 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ebef61_196f_4e79_adf1_20bea663abc3.slice/crio-81ab04d39afaf9b577c4b0575ebe7fa19889db8b17d6d75608f822d9bf1cf702 WatchSource:0}: Error finding container 81ab04d39afaf9b577c4b0575ebe7fa19889db8b17d6d75608f822d9bf1cf702: Status 404 returned error can't find the container with id 81ab04d39afaf9b577c4b0575ebe7fa19889db8b17d6d75608f822d9bf1cf702 Oct 30 17:12:12.563848 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.563829 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnkr\" (UniqueName: \"kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.563948 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.563853 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57bx\" (UniqueName: \"kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.563948 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.563833 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpsx\" (UniqueName: \"kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.626838 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.626796 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.629993 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.629975 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:12.636802 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.636786 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:12.638448 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.638421 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded427dd8_842e_4b41_9436_5e02ed499871.slice/crio-651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d WatchSource:0}: Error finding container 651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d: Status 404 returned error can't find the container with id 651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d Oct 30 17:12:12.639087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.639166 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdv9\" (UniqueName: \"kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.639166 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.639255 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639188 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.639255 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639217 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.639648 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.639620 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.640280 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.640253 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.640332 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.640293 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.640332 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.640305 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.643897 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.643876 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e85761_926d_4b83_9df4_e5be2b5844c4.slice/crio-405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7 WatchSource:0}: Error finding container 405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7: Status 404 returned error can't find the container with id 405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7 Oct 30 17:12:12.647228 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.647216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-77wmf" Oct 30 17:12:12.654961 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.654940 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37eb0a41_d5a7_4c71_ac24_e7e6a8bfda3a.slice/crio-9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c WatchSource:0}: Error finding container 9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c: Status 404 returned error can't find the container with id 9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c Oct 30 17:12:12.662281 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.662252 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdv9\" (UniqueName: \"kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.727689 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.727608 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.779466 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.779417 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:12.788385 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:12.788361 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd078773f_7b19_4a09_a258_a996048df5aa.slice/crio-503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4 WatchSource:0}: Error finding container 503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4: Status 404 returned error can't find the container with id 503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4 Oct 30 17:12:12.827874 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.827846 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.841194 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.841168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:12.841351 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.841199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:12.841351 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.841305 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.841448 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.841309 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.842315 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842297 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.842389 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842328 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.842389 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842338 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.842389 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842301 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.842498 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842399 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.842498 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.842410 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.842498 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842445 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:12.842498 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842484 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:13.842471073 +0000 UTC m=+15.607039875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:12.842631 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842502 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:12.842631 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842513 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:12.842631 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842522 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:12.842631 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.842552 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:13.842541338 +0000 UTC m=+15.607110128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:12.928234 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.928189 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:12.972891 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.972863 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/0.log" Oct 30 17:12:12.973365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.972910 2563 generic.go:356] "Generic (PLEG): container finished" podID="01c50f6fe0fd9f68f1ab538d873faa96" containerID="7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655" exitCode=1 Oct 30 17:12:12.973365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.972996 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerDied","Data":"7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655"} Oct 30 17:12:12.973365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.973158 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:12.976000 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.975978 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:12.976133 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.976012 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:12.976133 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.976027 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:12.976358 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:12.976333 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:12.977127 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.977075 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4gnb" event={"ID":"ed427dd8-842e-4b41-9436-5e02ed499871","Type":"ContainerStarted","Data":"651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d"} Oct 30 17:12:12.979785 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.979698 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerStarted","Data":"9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827"} Oct 30 17:12:12.987052 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.987026 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgdc6" event={"ID":"d078773f-7b19-4a09-a258-a996048df5aa","Type":"ContainerStarted","Data":"503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4"} Oct 30 17:12:12.988517 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.988492 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-77wmf" event={"ID":"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a","Type":"ContainerStarted","Data":"9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c"} Oct 30 17:12:12.990023 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.989998 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" event={"ID":"40e85761-926d-4b83-9df4-e5be2b5844c4","Type":"ContainerStarted","Data":"405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7"} Oct 30 17:12:12.994144 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.994120 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" event={"ID":"40ebef61-196f-4e79-adf1-20bea663abc3","Type":"ContainerStarted","Data":"81ab04d39afaf9b577c4b0575ebe7fa19889db8b17d6d75608f822d9bf1cf702"} Oct 30 17:12:12.995418 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.995395 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rzvbg" event={"ID":"aadcc9ba-2535-48cd-abef-1535e15f6ec6","Type":"ContainerStarted","Data":"a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b"} Oct 30 17:12:12.996722 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.996657 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m72vs" event={"ID":"26243dda-d8a0-457d-8844-81a32a6def00","Type":"ContainerStarted","Data":"245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c"} Oct 30 17:12:12.999061 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:12.999035 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7"} Oct 30 17:12:13.029056 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.028392 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.042889 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.042857 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:13.043055 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.042995 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.047847 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.044342 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.047847 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.044377 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.047847 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.044393 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.047847 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.044518 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:13.047847 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.044571 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:14.044552482 +0000 UTC m=+15.809121293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:13.129454 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.129398 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.230095 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.230017 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.330862 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.330828 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.431727 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.431691 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.531889 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.531810 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.632815 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.632782 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.733811 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.733775 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.834379 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.834328 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.850330 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.849994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:13.850330 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.850041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:13.850330 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.850166 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.850628 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.850421 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.853054 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853030 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.853172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853065 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.853172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853078 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.853172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853104 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.853172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853137 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.853172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.853155 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853249 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853301 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853318 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:15.853298179 +0000 UTC m=+17.617866987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853318 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853342 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:13.853429 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.853375 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:15.853365218 +0000 UTC m=+17.617934022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:13.935005 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.934965 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:13.938320 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938295 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:13.938458 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938337 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.938562 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938528 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:13.938562 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938555 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.938686 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:13.938686 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.938626 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:13.943433 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943408 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.943433 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943437 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.943590 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943451 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.943590 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943478 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.943590 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943512 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.943590 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943526 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.943759 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.943681 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:13.943759 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943743 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:13.943857 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943763 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:13.943857 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:13.943776 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:13.943857 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.943771 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:13.943987 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:13.943944 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:14.031596 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.031563 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/0.log" Oct 30 17:12:14.032120 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.031639 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8"} Oct 30 17:12:14.032120 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.031785 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:14.033292 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.033256 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:14.033423 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.033308 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:14.033423 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.033324 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:14.033645 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.033620 2563 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-147.ec2.internal\" not found" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:14.035249 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.035181 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.051996 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.051967 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:14.052149 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.052108 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:14.059867 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.059845 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:14.059999 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.059879 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:14.059999 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:14.059895 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:14.060114 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.060033 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:14.060114 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.060097 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:16.060076697 +0000 UTC m=+17.824645499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:14.135558 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.135472 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.235630 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.235590 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.336701 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.336662 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.437770 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.437693 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.538638 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.538601 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.639421 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.639389 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.732079 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.732041 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-lr6fs_openshift-multus(bbc52026-7904-40fd-95ec-1c7b3db1b5cf): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:14.732214 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.732092 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" podUID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" Oct 30 17:12:14.739651 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.739623 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.840385 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.840345 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:14.940947 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:14.940920 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.038004 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.037977 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.041143 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.041117 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.041454 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.041437 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.041524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.041471 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.041524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.041488 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.047545 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.047516 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-lr6fs_openshift-multus(bbc52026-7904-40fd-95ec-1c7b3db1b5cf): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:15.047643 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.047563 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" podUID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" Oct 30 17:12:15.142449 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.142403 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.161447 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.160541 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.167133 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.166308 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.167133 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.166345 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.167133 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.166360 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.244696 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.244632 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.345029 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.344934 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.445817 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.445761 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.546846 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.546791 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.647557 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.647478 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.748603 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.748551 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.849379 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.849321 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:15.867629 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.867149 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:15.867629 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.867196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:15.867629 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.867337 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.867629 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.867542 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.870378 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.870412 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.870426 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.870556 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.870615 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:19.870585655 +0000 UTC m=+21.635154450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.871212 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.871232 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.871246 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.871386 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.871400 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.871411 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:15.871541 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.871451 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:19.871436562 +0000 UTC m=+21.636005354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:15.921050 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.920864 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-rzvbg_kube-system(aadcc9ba-2535-48cd-abef-1535e15f6ec6): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:15.921050 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.920912 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="kube-system/konnectivity-agent-rzvbg" podUID="aadcc9ba-2535-48cd-abef-1535e15f6ec6" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.937983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.938036 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.938216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.938237 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.938285 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:15.938425 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.938323 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.941745 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.941777 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.941793 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.942034 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942140 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942161 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942174 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.942337 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942403 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942420 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:15.942433 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:15.942616 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.942548 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:15.949491 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:15.949436 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.040658 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.040428 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:16.043362 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.043336 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:16.043484 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.043374 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:16.043484 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.043388 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:16.048489 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.048463 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-rzvbg_kube-system(aadcc9ba-2535-48cd-abef-1535e15f6ec6): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:16.048602 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.048505 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="kube-system/konnectivity-agent-rzvbg" podUID="aadcc9ba-2535-48cd-abef-1535e15f6ec6" Oct 30 17:12:16.049981 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.049953 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.068774 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.068727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:16.068917 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.068860 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:16.072057 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.072031 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:16.072156 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.072069 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:16.072156 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.072085 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:16.072255 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.072204 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:16.072334 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.072256 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:20.072235357 +0000 UTC m=+21.836804162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:16.150939 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.150902 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.170013 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.169937 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:16.171507 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.171482 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:16.171621 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.171519 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:16.171621 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.171532 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:16.252044 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.251988 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.352351 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.352317 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.453377 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.453292 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.553463 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.553429 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.653906 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.653851 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.664728 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.664445 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:16.666905 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.666376 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:16.666905 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.666403 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:16.666905 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:16.666421 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:16.755067 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.754992 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.855390 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.855331 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:16.956490 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:16.956457 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.057502 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.057451 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.158364 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.158316 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.259172 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.259122 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.359600 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.359514 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.460688 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.460641 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.562138 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.561241 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.562138 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.561813 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="container csi-driver start failed in pod aws-ebs-csi-driver-node-v8ghl_openshift-cluster-csi-drivers(40e85761-926d-4b83-9df4-e5be2b5844c4): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:17.628971 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.628933 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="container node-ca start failed in pod node-ca-bgdc6_openshift-image-registry(d078773f-7b19-4a09-a258-a996048df5aa): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:17.628971 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.628969 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="openshift-image-registry/node-ca-bgdc6" podUID="d078773f-7b19-4a09-a258-a996048df5aa" Oct 30 17:12:17.661382 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.661337 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.678465 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.678415 2563 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:17.762293 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.762236 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.863030 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.862978 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.937878 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.937930 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.938173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.938202 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.938242 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:17.938524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.938305 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.939953 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.939983 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.939997 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.940197 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.940834 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.940859 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.940872 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.941047 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.941157 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.941175 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:17.941189 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:17.941408 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.941345 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:17.963302 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:17.963243 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:18.044585 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.044343 2563 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:18.047099 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.045966 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:18.047099 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.046013 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:18.047099 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.046027 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:18.050526 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:18.050226 2563 kuberuntime_manager.go:1707] "Unhandled Error" err="container node-ca start failed in pod node-ca-bgdc6_openshift-image-registry(d078773f-7b19-4a09-a258-a996048df5aa): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError" Oct 30 17:12:18.050526 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:18.050279 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="openshift-image-registry/node-ca-bgdc6" podUID="d078773f-7b19-4a09-a258-a996048df5aa" Oct 30 17:12:18.063504 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:18.063468 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:18.164100 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:18.164069 2563 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:18.170680 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.170653 2563 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:18.256365 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.256105 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:18.276387 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.276037 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:18.276387 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.276131 2563 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:18.277401 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.276739 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"] Oct 30 17:12:18.301560 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.301527 2563 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:18.323003 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.322967 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal"] Oct 30 17:12:18.323711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.323692 2563 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:18.369096 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.369071 2563 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:18.768203 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.767573 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:12:18.841229 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.841113 2563 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:12:18.841229 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:18.841200 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c8gkq" Oct 30 17:12:19.007780 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.007582 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" podStartSLOduration=1.007560308 podStartE2EDuration="1.007560308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:19.007466418 +0000 UTC m=+20.772035258" watchObservedRunningTime="2026-10-30 17:12:19.007560308 +0000 UTC m=+20.772129122" Oct 30 17:12:19.049730 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.049628 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" event={"ID":"40e85761-926d-4b83-9df4-e5be2b5844c4","Type":"ContainerStarted","Data":"b7e9537bc5b78abf74373c2bcd4086e197270ad4d758e50dcdff9098e83e5864"} Oct 30 17:12:19.888941 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.888687 2563 ???:1] "http: TLS handshake error from 10.0.139.222:60848: no serving certificate available for the kubelet" Oct 30 17:12:19.905918 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.905890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:19.906029 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.905934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:19.906102 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906057 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:19.906102 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906067 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:19.906102 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906088 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:19.906102 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906100 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:19.906315 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906120 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:27.906105223 +0000 UTC m=+29.670674012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:19.906315 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.906149 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:27.906132697 +0000 UTC m=+29.670701490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:19.938754 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.938672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:19.938927 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.938823 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:19.938927 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.938881 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:19.939043 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.938941 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:19.939043 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:19.938986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:19.939144 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:19.939054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:20.006920 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:20.006862 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-driver\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" podUID="40e85761-926d-4b83-9df4-e5be2b5844c4" Oct 30 17:12:20.054721 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.054686 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-77wmf" event={"ID":"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a","Type":"ContainerStarted","Data":"d4f1821def4f378e6d25b9e9fdb3475b63b366945657b491ac054bf8bacecc23"} Oct 30 17:12:20.056600 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.056568 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" event={"ID":"40e85761-926d-4b83-9df4-e5be2b5844c4","Type":"ContainerStarted","Data":"e5d651f2a2c9d576da6c9efd1cd015c1ce54a4f7aa475e4bb1a8d2523405d17b"} Oct 30 17:12:20.058926 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.058902 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m72vs" event={"ID":"26243dda-d8a0-457d-8844-81a32a6def00","Type":"ContainerStarted","Data":"12bbc671b93beb39b3d162942d56003ea9ddd7e78337fc6f067b8cef21e386f9"} Oct 30 17:12:20.107094 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.107059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:20.107283 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:20.107184 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:20.107283 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:20.107240 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:28.107222136 +0000 UTC m=+29.871790939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:20.107774 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.107727 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-77wmf" podStartSLOduration=0.993036737 podStartE2EDuration="8.10771014s" totalImagesPullingTime="7.114673403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:20.107170822 +0000 UTC m=+21.871739635" watchObservedRunningTime="2026-10-30 17:12:20.10771014 +0000 UTC m=+21.872278951" Oct 30 17:12:20.143700 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.143607 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m72vs" podStartSLOduration=1.407495122 podStartE2EDuration="8.143589642s" totalImagesPullingTime="6.73609452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:20.142833769 +0000 UTC m=+21.907402580" watchObservedRunningTime="2026-10-30 17:12:20.143589642 +0000 UTC m=+21.908158459" Oct 30 17:12:20.886213 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.886180 2563 ???:1] "http: TLS handshake error from 10.0.135.170:47914: no serving certificate available for the kubelet" Oct 30 17:12:20.905343 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:20.905311 2563 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:12:21.063661 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.063614 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" event={"ID":"40e85761-926d-4b83-9df4-e5be2b5844c4","Type":"ContainerStarted","Data":"b2a7dd1406e9c43ef1f26caa7e2dc6f4f38242a717cd5fba559b91d0e496e810"} Oct 30 17:12:21.834936 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.834833 2563 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:12:20.9053342Z","UUID":"6d9e3c4c-03f7-47e7-8e91-a159e8d71846","Handler":null,"Name":"","Endpoint":""} Oct 30 17:12:21.836526 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.836497 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:12:21.836526 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.836533 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:12:21.938047 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.938015 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:21.938640 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.938015 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:21.938640 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:21.938126 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:21.938640 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:21.938157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:21.938640 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:21.938248 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:21.938640 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:21.938367 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:22.067043 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:22.067010 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" event={"ID":"40ebef61-196f-4e79-adf1-20bea663abc3","Type":"ContainerStarted","Data":"454db6d5baa80334c62c53be595cca6948e547ebff5cd40ee12005391ca79286"} Oct 30 17:12:22.068524 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:22.068490 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w4gnb" event={"ID":"ed427dd8-842e-4b41-9436-5e02ed499871","Type":"ContainerStarted","Data":"fc6a5c36db24e3f9dab31133b5bb486d0a70d07fa144f63b85d50ea0cebdae08"} Oct 30 17:12:22.108669 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:22.108583 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" podStartSLOduration=1.2797110809999999 podStartE2EDuration="10.108569688s" totalImagesPullingTime="8.828858607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:22.10821612 +0000 UTC m=+23.872784962" watchObservedRunningTime="2026-10-30 17:12:22.108569688 +0000 UTC m=+23.873138497" Oct 30 17:12:22.108800 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:22.108685 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" podStartSLOduration=3.041028423 podStartE2EDuration="10.108680561s" totalImagesPullingTime="7.067652138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:21.10375001 +0000 UTC m=+22.868318823" watchObservedRunningTime="2026-10-30 17:12:22.108680561 +0000 UTC m=+23.873249571" Oct 30 17:12:22.143902 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:22.143855 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w4gnb" podStartSLOduration=3.103394118 podStartE2EDuration="10.143841157s" totalImagesPullingTime="7.040447039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:22.143630186 +0000 UTC m=+23.908199000" watchObservedRunningTime="2026-10-30 17:12:22.143841157 +0000 UTC m=+23.908409968" Oct 30 17:12:23.938112 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:23.938064 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:23.938112 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:23.938104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:23.938775 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:23.938088 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:23.938775 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:23.938191 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:23.938775 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:23.938273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:23.938775 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:23.938369 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:25.077473 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.077246 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/1.log" Oct 30 17:12:25.077966 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.077948 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/0.log" Oct 30 17:12:25.078006 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.077984 2563 generic.go:356] "Generic (PLEG): container finished" podID="01c50f6fe0fd9f68f1ab538d873faa96" containerID="d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8" exitCode=1 Oct 30 17:12:25.078039 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.078015 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerDied","Data":"d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8"} Oct 30 17:12:25.078105 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.078095 2563 scope.go:122] "RemoveContainer" containerID="7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655" Oct 30 17:12:25.078434 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.078411 2563 scope.go:122] "RemoveContainer" containerID="7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655" Oct 30 17:12:25.091702 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.091668 2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator_01c50f6fe0fd9f68f1ab538d873faa96_0 in pod sandbox faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 from index: no such id: '7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655'" containerID="7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655" Oct 30 17:12:25.091817 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.091720 2563 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator_01c50f6fe0fd9f68f1ab538d873faa96_0 in pod sandbox faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 from index: no such id: '7c4e85b47079f3df5937f9b2ab7b5909f8d12d0847f17c0630a94f025f354655'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator(01c50f6fe0fd9f68f1ab538d873faa96)\"" logger="UnhandledError" Oct 30 17:12:25.091975 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.091947 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator(01c50f6fe0fd9f68f1ab538d873faa96)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" podUID="01c50f6fe0fd9f68f1ab538d873faa96" Oct 30 17:12:25.938479 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.938283 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:25.938654 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.938318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:25.938654 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.938554 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:25.938654 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:25.938318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:25.938654 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.938624 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:25.938801 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:25.938696 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:26.082559 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082523 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"d703b8ea98edffa124431c1b8973d6e3fa412316f8dc35046c1413415c74eee0"} Oct 30 17:12:26.083065 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082598 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"6cd728f23812ff2555d9fb91db5d86b9a716915b0ab1110b02bccdebb08b7e45"} Oct 30 17:12:26.083065 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082629 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"45dd694275ee5b4b365015c64e031e79542b07ebecebe789c13ba2c4b5628c73"} Oct 30 17:12:26.083065 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082666 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"9b97b8c92088024c6d8df518d42a94d0322b831000c508f6a9663916a40850f5"} Oct 30 17:12:26.083065 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082693 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"357bd37e7eed99806085d2208ce67fdfaa787364e839b87fffa1f6282609ab2b"} Oct 30 17:12:26.083065 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.082730 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"e92b933719fd05b8471fd13db2b0c8c4a96237b06a0ef40a2ee652e21b86d352"} Oct 30 17:12:26.084055 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:26.084030 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/1.log" Oct 30 17:12:27.938002 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:27.937964 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:27.938434 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.938166 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:27.938434 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:27.938201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:27.938434 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.938301 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:27.938434 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:27.938333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:27.938434 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.938427 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:27.964965 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:27.964931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:27.965090 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:27.964981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:27.965143 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965086 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:27.965143 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965117 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:12:27.965143 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965131 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:12:27.965143 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965140 2563 projected.go:196] Error preparing data for projected volume kube-api-access-xh2ff for pod openshift-network-diagnostics/network-check-target-wlnpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:27.965563 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965149 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs podName:6d89b5f3-c0cb-4878-9496-768483a56be9 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:43.96513369 +0000 UTC m=+45.729702478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs") pod "network-metrics-daemon-lqgx2" (UID: "6d89b5f3-c0cb-4878-9496-768483a56be9") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:12:27.965563 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:27.965179 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff podName:e2f5736c-1133-4e5b-88b0-181bd447b670 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:43.965163957 +0000 UTC m=+45.729732747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xh2ff" (UniqueName: "kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff") pod "network-check-target-wlnpk" (UID: "e2f5736c-1133-4e5b-88b0-181bd447b670") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:12:28.096398 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:28.096359 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"136ef1e9983020a602b45cfa70de80d11964fd340c616454a4a9c1c2623eca56"} Oct 30 17:12:28.097697 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:28.097674 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="359574c158fad49c3369e8a87311c125fd066b3e8d22c4edfb3f00bc7d28fead" exitCode=0 Oct 30 17:12:28.097783 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:28.097705 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"359574c158fad49c3369e8a87311c125fd066b3e8d22c4edfb3f00bc7d28fead"} Oct 30 17:12:28.166630 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:28.166597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:28.166762 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:28.166691 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:28.166762 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:28.166738 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret podName:39f1686a-04e8-4f10-874a-51e855eea9a0 nodeName:}" failed. No retries permitted until 2026-10-30 17:12:44.166725022 +0000 UTC m=+45.931293811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret") pod "global-pull-secret-syncer-854ml" (UID: "39f1686a-04e8-4f10-874a-51e855eea9a0") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:12:29.938538 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:29.938513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:29.939098 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:29.938524 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:29.939098 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:29.938541 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:29.939098 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:29.938828 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:29.939098 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:29.938867 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:29.939098 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:29.938945 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:30.105970 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.105793 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" event={"ID":"6ee63db4-03fa-415b-9489-b188e6a71575","Type":"ContainerStarted","Data":"56a85c0bfc8fc5ab5037ebd6d418c784ec2fc000fb29365966b70fd3205954b1"} Oct 30 17:12:30.106127 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.106030 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:30.106127 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.106086 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:30.107732 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.107706 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="0f002132a6b5e1f820a9a06703144c0de123bcdbc07dea603606d7e5edcb4e07" exitCode=0 Oct 30 17:12:30.107854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.107771 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"0f002132a6b5e1f820a9a06703144c0de123bcdbc07dea603606d7e5edcb4e07"} Oct 30 17:12:30.109342 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.109312 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rzvbg" event={"ID":"aadcc9ba-2535-48cd-abef-1535e15f6ec6","Type":"ContainerStarted","Data":"6ede159a25c659faaf305bb8a34118559f81e40899e4c5335c8f4f1b182fb625"} Oct 30 17:12:30.121496 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.121472 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:30.121740 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.121725 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:30.186253 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.186213 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" podStartSLOduration=6.095610033 podStartE2EDuration="19.1862003s" totalImagesPullingTime="13.090590267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:30.185746314 +0000 UTC m=+31.950315125" watchObservedRunningTime="2026-10-30 17:12:30.1862003 +0000 UTC m=+31.950769111" Oct 30 17:12:30.220077 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:30.220033 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rzvbg" podStartSLOduration=14.838914848 podStartE2EDuration="18.220021322s" totalImagesPullingTime="3.381106474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:30.219909322 +0000 UTC m=+31.984478130" watchObservedRunningTime="2026-10-30 17:12:30.220021322 +0000 UTC m=+31.984590157" Oct 30 17:12:31.113793 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.113759 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="bd6a95c9fc3243897a09803a8131a0bc57ad237b00a170b0d986e81d05f31c56" exitCode=0 Oct 30 17:12:31.114192 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.113828 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"bd6a95c9fc3243897a09803a8131a0bc57ad237b00a170b0d986e81d05f31c56"} Oct 30 17:12:31.114192 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.113892 2563 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:31.938381 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.938351 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:31.938615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.938386 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:31.938615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:31.938357 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:31.938615 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:31.938448 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:31.938615 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:31.938520 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:31.938615 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:31.938573 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:32.118492 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.118463 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="0c06b6380aa4cbd9f2fdb5ad92f95fa74fffe10c8d7ade4ecfa5214766ce5974" exitCode=0 Oct 30 17:12:32.118908 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.118542 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"0c06b6380aa4cbd9f2fdb5ad92f95fa74fffe10c8d7ade4ecfa5214766ce5974"} Oct 30 17:12:32.120036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.120015 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgdc6" event={"ID":"d078773f-7b19-4a09-a258-a996048df5aa","Type":"ContainerStarted","Data":"b8fcc3e9c5a719c39c51b48ff8a9e7693d7a80649134861c0fbc7525b5207362"} Oct 30 17:12:32.120095 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.120068 2563 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:12:32.530586 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.530549 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:32.531240 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.531211 2563 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:32.566611 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:32.566549 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bgdc6" podStartSLOduration=15.728271617 podStartE2EDuration="20.566530696s" totalImagesPullingTime="4.838259079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:32.216800737 +0000 UTC m=+33.981369561" watchObservedRunningTime="2026-10-30 17:12:32.566530696 +0000 UTC m=+34.331099510" Oct 30 17:12:33.097708 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.097671 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:33.122337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.122307 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:33.122740 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.122634 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:33.937809 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.937783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:33.937809 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.937808 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:33.938000 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:33.937893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:33.938000 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:33.937899 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:33.938000 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:33.937957 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:33.938092 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:33.938031 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:34.126567 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:34.126535 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="7cab40f5fbb6d1a0b36e3420b3f4b13d0983c9628c219cff8a818cbac9f7b973" exitCode=0 Oct 30 17:12:34.126977 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:34.126604 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"7cab40f5fbb6d1a0b36e3420b3f4b13d0983c9628c219cff8a818cbac9f7b973"} Oct 30 17:12:35.131373 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:35.131340 2563 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="58f1644d7710daa5591e129e33a66212f4fe9e8e3691964caaee201eaeda39aa" exitCode=0 Oct 30 17:12:35.131838 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:35.131408 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerDied","Data":"58f1644d7710daa5591e129e33a66212f4fe9e8e3691964caaee201eaeda39aa"} Oct 30 17:12:35.938417 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:35.938380 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:35.938599 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:35.938393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:35.938599 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:35.938489 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:35.938599 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:35.938393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:35.938736 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:35.938555 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:35.938736 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:35.938650 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:36.136643 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:36.136611 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" event={"ID":"bbc52026-7904-40fd-95ec-1c7b3db1b5cf","Type":"ContainerStarted","Data":"211ea605818f43b7928fc14eda2bf81cdf53a55b3c0ee13c898a05a59271d106"} Oct 30 17:12:36.202917 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:36.202819 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lr6fs" podStartSLOduration=17.503886456 podStartE2EDuration="24.202803386s" totalImagesPullingTime="6.69891693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:36.192425118 +0000 UTC m=+37.956993960" watchObservedRunningTime="2026-10-30 17:12:36.202803386 +0000 UTC m=+37.967372178" Oct 30 17:12:36.613329 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:36.613291 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c8gkq" Oct 30 17:12:37.645892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:37.645840 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:36 +0000 UTC" deadline="2028-07-31 04:54:00.772931128 +0000 UTC" Oct 30 17:12:37.645892 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:37.645884 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15347h41m23.127050143s" Oct 30 17:12:37.938672 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:37.938438 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:37.938837 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:37.938477 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:37.938837 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:37.938716 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:37.938837 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:37.938538 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:37.938837 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:37.938818 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:37.938980 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:37.938848 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:38.159142 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.159110 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-854ml"] Oct 30 17:12:38.159308 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.159238 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:38.159399 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:38.159376 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:38.160359 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.160331 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lqgx2"] Oct 30 17:12:38.160468 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.160446 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:38.160575 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:38.160557 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:38.160807 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.160782 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlnpk"] Oct 30 17:12:38.160923 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.160869 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:38.160985 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:38.160948 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:38.646036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.646001 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:36 +0000 UTC" deadline="2028-04-14 16:51:51.026029749 +0000 UTC" Oct 30 17:12:38.646036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:38.646031 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12767h39m12.380001517s" Oct 30 17:12:39.148836 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.148656 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_01c50f6fe0fd9f68f1ab538d873faa96/setup/1.log" Oct 30 17:12:39.148965 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.148850 2563 generic.go:356] "Generic (PLEG): container finished" podID="01c50f6fe0fd9f68f1ab538d873faa96" containerID="2b1fc76f4ba5a972e7e737894e118a25856776b3c719c027b37f9e5732b7acc3" exitCode=0 Oct 30 17:12:39.148965 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.148885 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerDied","Data":"2b1fc76f4ba5a972e7e737894e118a25856776b3c719c027b37f9e5732b7acc3"} Oct 30 17:12:39.148965 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.148933 2563 scope.go:122] "RemoveContainer" containerID="d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8" Oct 30 17:12:39.149247 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.149231 2563 scope.go:122] "RemoveContainer" containerID="d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8" Oct 30 17:12:39.175841 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:39.175802 2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator_01c50f6fe0fd9f68f1ab538d873faa96_1 in pod sandbox faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 from index: no such id: 'd5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8'" containerID="d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8" Oct 30 17:12:39.175942 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.175853 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal_openshift-machine-config-operator_01c50f6fe0fd9f68f1ab538d873faa96_1 in pod sandbox faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521 from index: no such id: 'd5eee4a05ca580ec7c0b94e4e018fe6e738dd5c8903b60213e3d845874b0c6b8'" Oct 30 17:12:39.937905 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.937869 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:39.938659 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.937949 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:39.938659 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:39.937967 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:39.938659 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:39.938062 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:39.938659 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:39.938169 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:39.938659 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:39.938229 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:40.152594 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:40.152563 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"2a68b8bf2a646a982a556da4356f3660b4aeca39aa52a4cd2f50bbdc5d0c571f"} Oct 30 17:12:40.188201 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:40.188105 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" podStartSLOduration=22.188090918 podStartE2EDuration="22.188090918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:40.187886874 +0000 UTC m=+41.952455684" watchObservedRunningTime="2026-10-30 17:12:40.188090918 +0000 UTC m=+41.952659728" Oct 30 17:12:40.396879 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:40.396849 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m72vs_26243dda-d8a0-457d-8844-81a32a6def00/dns-node-resolver/0.log" Oct 30 17:12:41.531631 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:41.531562 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgdc6_d078773f-7b19-4a09-a258-a996048df5aa/node-ca/0.log" Oct 30 17:12:41.937946 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:41.937910 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:41.938132 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:41.938012 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:41.938132 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:41.938023 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:41.938132 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:41.938022 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wlnpk" podUID="e2f5736c-1133-4e5b-88b0-181bd447b670" Oct 30 17:12:41.938132 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:41.938096 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-854ml" podUID="39f1686a-04e8-4f10-874a-51e855eea9a0" Oct 30 17:12:41.938323 ip-10-0-143-147 kubenswrapper[2563]: E1030 17:12:41.938163 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lqgx2" podUID="6d89b5f3-c0cb-4878-9496-768483a56be9" Oct 30 17:12:43.485816 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.485746 2563 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeReady" Oct 30 17:12:43.486186 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.485856 2563 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:12:43.540668 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.540634 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:12:43.561574 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.561543 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tzxkz"] Oct 30 17:12:43.561762 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.561675 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.575494 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.575470 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:12:43.575494 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.575482 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:12:43.575494 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.575472 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:12:43.575732 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.575716 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-98bcg"] Oct 30 17:12:43.575826 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.575814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.578357 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.578302 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:12:43.593531 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.593508 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:12:43.593531 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.593540 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98bcg"] Oct 30 17:12:43.593726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.593617 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.593726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.593633 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzxkz"] Oct 30 17:12:43.648723 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.648686 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9628"] Oct 30 17:12:43.661566 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.661530 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.674478 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.674621 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674483 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.674621 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674522 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8fg\" (UniqueName: \"kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.674621 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.674726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.674726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674677 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.674726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.674726 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.674843 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllp8\" (UniqueName: \"kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.674843 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.674901 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.674929 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxlxl\" (UniqueName: \"kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.674960 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.674992 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674963 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.674992 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.674981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.675075 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.675075 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675062 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.675170 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.675219 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.675868 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675840 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:12:43.676004 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675989 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:12:43.676004 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675995 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:12:43.676129 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676003 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:12:43.676129 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.675999 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\"" Oct 30 17:12:43.676129 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676024 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:12:43.676129 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676012 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:12:43.676129 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676119 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\"" Oct 30 17:12:43.676372 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676189 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:12:43.676372 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676253 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:12:43.676372 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676333 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:12:43.676527 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.676513 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vqnr7\"" Oct 30 17:12:43.677363 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.677337 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9628"] Oct 30 17:12:43.687736 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.687714 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:12:43.725666 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.725636 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:12:43.725849 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.725820 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\"" Oct 30 17:12:43.776087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776016 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:12:43.776087 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776017 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxlxl\" (UniqueName: \"kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776337 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776544 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776586 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8fg\" (UniqueName: \"kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.776711 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.777124 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776713 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.777124 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.776735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.777234 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777127 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.777541 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777520 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.777610 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.777610 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllp8\" (UniqueName: \"kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.777714 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777689 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.777767 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777721 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.777767 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777749 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.777942 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777924 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.777998 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.777959 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.778036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.778009 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.780804 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.780781 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.780909 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.780805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.780909 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.780805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.780909 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.780890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.781036 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.781008 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.799645 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.799618 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.799763 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.799643 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8fg\" (UniqueName: \"kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.799763 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.799669 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllp8\" (UniqueName: \"kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.799854 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.799833 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxlxl\" (UniqueName: \"kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:43.894150 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.894110 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:43.935764 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.935744 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:43.937965 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.937933 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:43.938056 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.937972 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:43.938107 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.938078 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:43.951441 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951409 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:12:43.951441 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951425 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:12:43.951615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951521 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\"" Oct 30 17:12:43.951615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951524 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\"" Oct 30 17:12:43.951615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951550 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:12:43.951615 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951579 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\"" Oct 30 17:12:43.951732 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.951629 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:12:43.969234 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.969201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:43.978923 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.978886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:43.979163 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.979142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:43.981596 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.981570 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:43.981750 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.981730 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:43.987445 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:43.987419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:44.034182 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.033478 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:44.180934 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.180898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:44.181935 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.181909 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv"] Oct 30 17:12:44.184064 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.184030 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:44.199214 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.199192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:44.205642 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.205618 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv"] Oct 30 17:12:44.212163 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.212131 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 17:12:44.212163 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.212148 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5c2r9\"" Oct 30 17:12:44.271918 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.271883 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:44.279527 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.279450 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tzxkz"] Oct 30 17:12:44.282381 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.281858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bk4wv\" (UID: \"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:44.282693 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.282675 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:12:44.285908 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.285872 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c23856e_00f0_409e_a47b_6754355fd5ce.slice/crio-fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d WatchSource:0}: Error finding container fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d: Status 404 returned error can't find the container with id fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d Oct 30 17:12:44.287393 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.287368 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6723eff8_01c0_484c_88eb_7f363b0e565a.slice/crio-c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4 WatchSource:0}: Error finding container c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4: Status 404 returned error can't find the container with id c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4 Oct 30 17:12:44.291549 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.291514 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:44.294545 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.294526 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98bcg"] Oct 30 17:12:44.299127 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.299101 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320a0cd8_4ba1_4247_b0c7_3cfc8876b471.slice/crio-3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35 WatchSource:0}: Error finding container 3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35: Status 404 returned error can't find the container with id 3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35 Oct 30 17:12:44.327019 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.326995 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9628"] Oct 30 17:12:44.334021 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.333990 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffacaf7_208e_4092_9aaf_3adba31e8934.slice/crio-812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d WatchSource:0}: Error finding container 812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d: Status 404 returned error can't find the container with id 812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d Oct 30 17:12:44.366154 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.366128 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lqgx2"] Oct 30 17:12:44.382461 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.382182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bk4wv\" (UID: \"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:44.384824 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.384801 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bk4wv\" (UID: \"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:44.519617 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.519579 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:44.586400 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.586338 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wlnpk"] Oct 30 17:12:44.592632 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.592593 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f5736c_1133_4e5b_88b0_181bd447b670.slice/crio-b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8 WatchSource:0}: Error finding container b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8: Status 404 returned error can't find the container with id b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8 Oct 30 17:12:44.615923 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.615893 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-854ml"] Oct 30 17:12:44.620483 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.620457 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f1686a_04e8_4f10_874a_51e855eea9a0.slice/crio-965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d WatchSource:0}: Error finding container 965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d: Status 404 returned error can't find the container with id 965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d Oct 30 17:12:44.831172 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:44.831082 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv"] Oct 30 17:12:44.836175 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:44.836146 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909cd2de_faa4_4b8f_a0b1_5bc4e261b32c.slice/crio-bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3 WatchSource:0}: Error finding container bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3: Status 404 returned error can't find the container with id bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3 Oct 30 17:12:45.166080 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.165998 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-854ml" event={"ID":"39f1686a-04e8-4f10-874a-51e855eea9a0","Type":"ContainerStarted","Data":"965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d"} Oct 30 17:12:45.167900 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.167769 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" event={"ID":"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c","Type":"ContainerStarted","Data":"bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3"} Oct 30 17:12:45.170143 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.170116 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98bcg" event={"ID":"320a0cd8-4ba1-4247-b0c7-3cfc8876b471","Type":"ContainerStarted","Data":"3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35"} Oct 30 17:12:45.172625 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.172599 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd6766d5-btnxw" event={"ID":"6723eff8-01c0-484c-88eb-7f363b0e565a","Type":"ContainerStarted","Data":"c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4"} Oct 30 17:12:45.174381 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.174353 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9628" event={"ID":"affacaf7-208e-4092-9aaf-3adba31e8934","Type":"ContainerStarted","Data":"812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d"} Oct 30 17:12:45.176295 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.176241 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlnpk" event={"ID":"e2f5736c-1133-4e5b-88b0-181bd447b670","Type":"ContainerStarted","Data":"b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8"} Oct 30 17:12:45.179363 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.179301 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lqgx2" event={"ID":"6d89b5f3-c0cb-4878-9496-768483a56be9","Type":"ContainerStarted","Data":"4fd390e02b7b804d0df8c7923cca93fc5a9734c92d0b5a1dc7611cb4d4671840"} Oct 30 17:12:45.183147 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.183054 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzxkz" event={"ID":"4c23856e-00f0-409e-a47b-6754355fd5ce","Type":"ContainerStarted","Data":"789b238233c75b00255510e91f00fd62dc883ebb47e96cbb84c4cb4274e46884"} Oct 30 17:12:45.183147 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:45.183105 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzxkz" event={"ID":"4c23856e-00f0-409e-a47b-6754355fd5ce","Type":"ContainerStarted","Data":"fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d"} Oct 30 17:12:47.193825 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:47.193771 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lqgx2" event={"ID":"6d89b5f3-c0cb-4878-9496-768483a56be9","Type":"ContainerStarted","Data":"e0097be444194b6048ff5850e957afc729d95ab74f97d2403df908a1147f19ce"} Oct 30 17:12:47.194247 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:47.193841 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lqgx2" event={"ID":"6d89b5f3-c0cb-4878-9496-768483a56be9","Type":"ContainerStarted","Data":"65fb83726f668c09d4e03e3931398077b9044e75366a06f316f8ca0d5b27e784"} Oct 30 17:12:47.199483 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:47.199412 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzxkz" event={"ID":"4c23856e-00f0-409e-a47b-6754355fd5ce","Type":"ContainerStarted","Data":"439b5fec8806d450c0de88a5bc17151816ec53d757bef3de3b18f7d8022a670b"} Oct 30 17:12:47.235558 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:47.235337 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lqgx2" podStartSLOduration=32.91692453 podStartE2EDuration="35.235313487s" totalImagesPullingTime="2.318388957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:47.232801484 +0000 UTC m=+48.997370295" watchObservedRunningTime="2026-10-30 17:12:47.235313487 +0000 UTC m=+48.999882324" Oct 30 17:12:48.206015 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:48.205984 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98bcg" event={"ID":"320a0cd8-4ba1-4247-b0c7-3cfc8876b471","Type":"ContainerStarted","Data":"5c1f3e95086f807ce798e008f9ca540b64288e2f2a1b59bd527d6d51d2041e35"} Oct 30 17:12:48.206522 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:48.206035 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98bcg" event={"ID":"320a0cd8-4ba1-4247-b0c7-3cfc8876b471","Type":"ContainerStarted","Data":"fdf5446f58aa2772512487f23f14f35af8d7d5a16218ac061b1316f43063cd7f"} Oct 30 17:12:48.206522 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:48.206082 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:48.245088 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:48.245001 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-98bcg" podStartSLOduration=2.070647882 podStartE2EDuration="5.244984148s" totalImagesPullingTime="3.174336266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:48.244064697 +0000 UTC m=+50.008633505" watchObservedRunningTime="2026-10-30 17:12:48.244984148 +0000 UTC m=+50.009552957" Oct 30 17:12:49.211168 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:49.211128 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" event={"ID":"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c","Type":"ContainerStarted","Data":"df7bd009c050023cec8f367f3f432c8484add7698a0580d3d5e64448758633f0"} Oct 30 17:12:49.211663 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:49.211531 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:49.218089 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:49.218065 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:49.286218 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:49.286158 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" podStartSLOduration=1.5908345179999999 podStartE2EDuration="5.28613649s" totalImagesPullingTime="3.695301972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:49.248559427 +0000 UTC m=+51.013128239" watchObservedRunningTime="2026-10-30 17:12:49.28613649 +0000 UTC m=+51.050705396" Oct 30 17:12:50.216710 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.216406 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd6766d5-btnxw" event={"ID":"6723eff8-01c0-484c-88eb-7f363b0e565a","Type":"ContainerStarted","Data":"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4"} Oct 30 17:12:50.218288 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.218234 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9628" event={"ID":"affacaf7-208e-4092-9aaf-3adba31e8934","Type":"ContainerStarted","Data":"9912f04870b08e2493c6796f67737c19c931cf953ef53e91c0cbb381ecb7e515"} Oct 30 17:12:50.221442 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.221406 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tzxkz" event={"ID":"4c23856e-00f0-409e-a47b-6754355fd5ce","Type":"ContainerStarted","Data":"f285c5d96dcda63efd726afc937899348f00740dcaf36bdd93ad531cf93be503"} Oct 30 17:12:50.257429 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.257332 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58cd6766d5-btnxw" podStartSLOduration=1.547450561 podStartE2EDuration="7.257313851s" totalImagesPullingTime="5.70986329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:50.256754126 +0000 UTC m=+52.021322948" watchObservedRunningTime="2026-10-30 17:12:50.257313851 +0000 UTC m=+52.021882663" Oct 30 17:12:50.300654 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.299809 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tzxkz" podStartSLOduration=2.058019318 podStartE2EDuration="7.299787182s" totalImagesPullingTime="5.241767864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:50.298941492 +0000 UTC m=+52.063510303" watchObservedRunningTime="2026-10-30 17:12:50.299787182 +0000 UTC m=+52.064355994" Oct 30 17:12:50.337169 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.337113 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9628" podStartSLOduration=2.22317438 podStartE2EDuration="7.337092047s" totalImagesPullingTime="5.113917667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:50.336064642 +0000 UTC m=+52.100633453" watchObservedRunningTime="2026-10-30 17:12:50.337092047 +0000 UTC m=+52.101660861" Oct 30 17:12:50.392327 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.392293 2563 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-svgft"] Oct 30 17:12:50.417362 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.417339 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.418919 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.418894 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-svgft"] Oct 30 17:12:50.430821 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.430752 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 17:12:50.430821 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.430760 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:12:50.430821 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.430812 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:12:50.431081 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.430924 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 17:12:50.431081 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.431008 2563 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlm27\"" Oct 30 17:12:50.431081 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.431072 2563 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:12:50.530740 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.530656 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.530904 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.530777 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.530957 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.530908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xwb\" (UniqueName: \"kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.531002 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.530960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.632326 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.632292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xwb\" (UniqueName: \"kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.632525 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.632350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.632590 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.632548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.632649 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.632605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.633399 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.633373 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.637186 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.637058 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.637324 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.637063 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.654703 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.654676 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xwb\" (UniqueName: \"kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:50.774864 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:50.774831 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:51.100121 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:51.100029 2563 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-svgft"] Oct 30 17:12:51.104614 ip-10-0-143-147 kubenswrapper[2563]: W1030 17:12:51.104589 2563 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f87ebf_74d3_4c17_aa6a_c8dff6f9e08b.slice/crio-d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6 WatchSource:0}: Error finding container d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6: Status 404 returned error can't find the container with id d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6 Oct 30 17:12:51.227340 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:51.226897 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wlnpk" event={"ID":"e2f5736c-1133-4e5b-88b0-181bd447b670","Type":"ContainerStarted","Data":"76217c9ce51e0577a8d959392d273004ddb855783ac47bd6b1ad727030e548bb"} Oct 30 17:12:51.227340 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:51.227057 2563 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:51.228632 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:51.228603 2563 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" event={"ID":"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b","Type":"ContainerStarted","Data":"d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6"} Oct 30 17:12:51.263945 ip-10-0-143-147 kubenswrapper[2563]: I1030 17:12:51.263866 2563 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wlnpk" podStartSLOduration=34.490030591 podStartE2EDuration="40.263852504s" totalImagesPullingTime="5.773821913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:51.262972142 +0000 UTC m=+53.027540953" watchObservedRunningTime="2026-10-30 17:12:51.263852504 +0000 UTC m=+53.028421312" Oct 30 17:12:52.166805 ip-10-0-143-147 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:12:52.174037 ip-10-0-143-147 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:12:52.174230 ip-10-0-143-147 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:12:52.174462 ip-10-0-143-147 systemd[1]: kubelet.service: Consumed 3.056s CPU time. Oct 30 17:12:52.184889 ip-10-0-143-147 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:12:52.257979 ip-10-0-143-147 kubenswrapper[4622]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:52.257979 ip-10-0-143-147 kubenswrapper[4622]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:12:52.257979 ip-10-0-143-147 kubenswrapper[4622]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:52.257979 ip-10-0-143-147 kubenswrapper[4622]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:12:52.260752 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260648 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.260752 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260751 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.260752 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260756 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.260752 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260761 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260766 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260771 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260775 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260781 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260786 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260791 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260795 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260799 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260804 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260809 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260813 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260818 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260831 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260835 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260838 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260841 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260843 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260847 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260850 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.261010 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260853 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260855 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260859 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260861 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260864 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260867 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260870 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260872 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260875 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260879 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260882 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260885 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260887 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260890 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260893 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260895 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260898 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260901 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260903 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.261878 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260906 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260909 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260911 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260914 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260917 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260920 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260922 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260925 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260928 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260931 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260935 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260940 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260943 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260947 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260950 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260953 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260956 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260958 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260961 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.262719 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260965 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260968 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260971 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260974 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260977 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260979 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260983 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260986 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260989 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260992 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260994 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260997 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.260999 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261003 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261007 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261012 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261016 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261021 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261024 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.263542 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261027 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261030 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261032 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261035 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261038 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261041 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261044 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261046 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261049 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261052 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261054 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261057 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261060 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261064 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261067 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261071 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261073 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261076 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261080 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261083 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.264374 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261085 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261088 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261091 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261094 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261096 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.261101 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262780 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262791 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262795 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262798 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262801 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262804 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262807 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262810 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262813 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262817 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262821 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262826 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262830 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.264990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262834 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262837 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262840 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262843 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262846 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262849 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262852 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262855 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262858 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262861 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262863 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262866 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262870 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262872 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262875 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262878 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262881 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262883 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262886 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262889 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.265884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262892 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262895 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262898 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262900 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262903 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262905 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262908 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262911 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262913 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262916 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262919 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262923 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262927 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262932 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262936 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262940 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262944 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262948 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262953 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262957 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.266763 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262960 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262964 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262968 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262972 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262975 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262979 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262982 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262985 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262989 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262993 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.262997 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263001 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263005 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263009 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263013 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263018 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263022 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263026 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263030 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263037 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.267615 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263042 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263044 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263047 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263050 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263053 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263055 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263058 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263061 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263063 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263066 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263070 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263072 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263075 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263078 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263080 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263084 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263088 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263090 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263095 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.268477 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263101 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263107 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263114 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263118 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263121 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263124 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263126 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263129 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263132 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263135 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263138 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263141 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263146 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263148 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263876 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263885 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263888 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263892 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.269296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263897 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263902 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263906 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263911 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263915 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263920 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.263925 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264006 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264011 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264016 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264021 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264025 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264030 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264041 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264046 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264052 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264057 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264061 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264065 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.270086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264068 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264072 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264076 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264079 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264083 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264086 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264090 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264098 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264103 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264107 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264111 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264115 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264119 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264125 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264129 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264132 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264136 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264141 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264145 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.270918 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264148 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264151 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264156 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264160 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264164 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264168 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264172 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264176 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264186 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264191 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264195 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264199 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264203 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264207 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264211 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264214 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264218 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264222 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264226 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264230 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.271794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264236 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264239 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264243 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264247 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264251 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264255 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264278 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264283 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264287 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264291 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264295 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264298 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264302 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264306 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264310 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264314 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264318 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264321 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264326 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264330 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.272661 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264334 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264339 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264343 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264346 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264350 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264354 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264358 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264361 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264366 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264369 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264373 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264376 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264382 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264386 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264390 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264395 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264399 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264403 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264407 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.273415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264410 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264414 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264419 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264423 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.264427 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264570 4622 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264586 4622 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264599 4622 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264630 4622 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264639 4622 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264644 4622 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264658 4622 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264664 4622 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264669 4622 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264673 4622 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264682 4622 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264687 4622 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264692 4622 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264696 4622 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264701 4622 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264705 4622 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264709 4622 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264713 4622 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264720 4622 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:12:52.274203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264724 4622 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264731 4622 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264736 4622 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264741 4622 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264746 4622 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264751 4622 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264755 4622 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264760 4622 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264764 4622 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264769 4622 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264775 4622 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264781 4622 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264785 4622 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264791 4622 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264796 4622 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264800 4622 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264804 4622 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264808 4622 flags.go:64] FLAG: --enable-server="true" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264813 4622 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264819 4622 flags.go:64] FLAG: --event-burst="100" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264824 4622 flags.go:64] FLAG: --event-qps="50" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264829 4622 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264834 4622 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264839 4622 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:12:52.275301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264846 4622 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264852 4622 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264857 4622 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264862 4622 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264867 4622 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264872 4622 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264876 4622 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264879 4622 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264882 4622 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264886 4622 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264890 4622 flags.go:64] FLAG: --feature-gates="" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264894 4622 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264897 4622 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264901 4622 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264905 4622 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264910 4622 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264915 4622 flags.go:64] FLAG: --help="false" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264920 4622 flags.go:64] FLAG: --hostname-override="ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264928 4622 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264933 4622 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264938 4622 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264944 4622 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264949 4622 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264954 4622 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:12:52.276387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264958 4622 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264963 4622 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264966 4622 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264969 4622 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264972 4622 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264975 4622 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264978 4622 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264982 4622 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264985 4622 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264989 4622 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264992 4622 flags.go:64] FLAG: --lock-file="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264995 4622 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.264998 4622 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265001 4622 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265009 4622 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265014 4622 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265019 4622 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265025 4622 flags.go:64] FLAG: --logging-format="text" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265030 4622 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265034 4622 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265038 4622 flags.go:64] FLAG: --manifest-url="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265041 4622 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265045 4622 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265049 4622 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265055 4622 flags.go:64] FLAG: --max-pods="110" Oct 30 17:12:52.277438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265060 4622 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265067 4622 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265072 4622 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265077 4622 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265080 4622 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265083 4622 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265086 4622 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265094 4622 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265097 4622 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265100 4622 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265104 4622 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265108 4622 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265113 4622 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265117 4622 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265122 4622 flags.go:64] FLAG: --port="10250" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265127 4622 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265132 4622 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd063e6ef3b21223" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265137 4622 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265142 4622 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265146 4622 flags.go:64] FLAG: --register-node="true" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265151 4622 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265157 4622 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265160 4622 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265163 4622 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265168 4622 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265172 4622 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:12:52.278373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265176 4622 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265179 4622 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265182 4622 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265185 4622 flags.go:64] FLAG: --runonce="false" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265188 4622 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265191 4622 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265194 4622 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265200 4622 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265203 4622 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265207 4622 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265212 4622 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265217 4622 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265222 4622 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265227 4622 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265231 4622 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265235 4622 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265240 4622 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265244 4622 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265249 4622 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265257 4622 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265275 4622 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265280 4622 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265288 4622 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265293 4622 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265298 4622 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:12:52.279479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265303 4622 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265307 4622 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265310 4622 flags.go:64] FLAG: --v="2" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265315 4622 flags.go:64] FLAG: --version="false" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265319 4622 flags.go:64] FLAG: --vmodule="" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265325 4622 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.265329 4622 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265552 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265558 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265563 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265569 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265575 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265581 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265586 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265592 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265597 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265601 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265606 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265611 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265614 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265616 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.280549 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265620 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265622 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265625 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265628 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265631 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265633 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265636 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265640 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265644 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265648 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265652 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265663 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265667 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265673 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265679 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265683 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265691 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265694 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265696 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.281469 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265699 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265702 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265705 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265708 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265710 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265713 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265718 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265721 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265724 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265726 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265729 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265731 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265734 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265737 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265740 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265743 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265745 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265748 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265752 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.282349 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265756 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265761 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265764 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265768 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265772 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265776 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265787 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265791 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265795 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265798 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265802 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265805 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265807 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265810 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265813 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265815 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265818 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265821 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265824 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.283178 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265828 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265831 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265833 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265837 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265842 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265846 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265850 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265854 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265857 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265861 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265865 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265869 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265873 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265877 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265881 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265884 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265887 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265890 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265893 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.284052 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265896 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265904 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265907 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265910 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265916 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265920 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265924 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265928 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265932 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265936 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265939 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265943 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265947 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265953 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265957 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.265960 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.284868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.266003 4622 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266050 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266055 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266058 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266062 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266067 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266072 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266077 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266082 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266087 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266092 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266097 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266101 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266105 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266108 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266111 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266113 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266116 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266120 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266123 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.285563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266125 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266128 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266131 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266134 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266137 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266140 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266142 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266145 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266148 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266152 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266155 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266159 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266163 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266168 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266172 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266176 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266180 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266184 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266189 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.286430 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266194 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266199 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266203 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266209 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266213 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266216 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266220 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266225 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266227 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266231 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266235 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266239 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266244 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266249 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266253 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266257 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266275 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266278 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266281 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266284 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.287226 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266286 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266289 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266293 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266296 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266299 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266301 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266304 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266308 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266312 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266317 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266322 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266328 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266332 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266336 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266341 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266346 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266351 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266356 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266360 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.288100 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266363 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266366 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266369 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266371 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266374 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266377 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266380 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266383 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266387 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266391 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266395 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266399 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266403 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266406 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266409 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266414 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266418 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266422 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266427 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266431 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.288911 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266435 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266440 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266445 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266449 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266453 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266456 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266460 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266464 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.266468 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.275612 4622 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.275635 4622 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.275658 4622 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.275666 4622 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275737 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275744 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275750 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275754 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275759 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275764 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.289882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275767 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275770 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275773 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275776 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275779 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275782 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275785 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275788 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275793 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275797 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275800 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275805 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275810 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275814 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275818 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275823 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275828 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275832 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275835 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.290698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275837 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275840 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275843 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275847 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275850 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275853 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275855 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275858 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275861 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275864 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275867 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275872 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275876 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275879 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275882 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275886 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275890 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275895 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275899 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275904 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.291645 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275906 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275909 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275911 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275915 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275918 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275920 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275925 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275929 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275934 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275938 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275942 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275946 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275951 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275955 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275960 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275964 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275969 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275974 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275978 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.292381 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275983 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275990 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.275995 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276000 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276005 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276009 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276014 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276018 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276022 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276026 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276030 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276043 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276048 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276052 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276055 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276058 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276061 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276064 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276067 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.292976 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276072 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276076 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276081 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276086 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276090 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276094 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276099 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276103 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276108 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276112 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276116 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276119 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276122 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276125 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276130 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276134 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276138 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276143 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276147 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276151 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.293740 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276155 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276159 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276166 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.276204 4622 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276424 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276432 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276438 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276442 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276445 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276448 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276452 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276456 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276461 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276465 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276469 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276473 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.294642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276477 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276481 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276486 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276490 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276495 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276498 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276501 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276505 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276509 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276514 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276518 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276523 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276527 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276531 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276536 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276541 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276545 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276551 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276556 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.295203 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276560 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276563 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276566 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276569 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276572 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276574 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276577 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276581 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276584 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276587 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276590 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276593 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276595 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276598 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276601 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276604 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276606 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276610 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276615 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.295979 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276619 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276624 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276629 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276634 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276638 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276643 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276646 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276649 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276651 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276654 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276657 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276660 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276662 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276665 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276668 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276670 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276673 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276677 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276681 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276684 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.296821 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276687 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276690 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276693 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276696 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276698 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276701 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276705 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276709 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276713 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276717 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276720 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276722 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276725 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276728 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276730 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276733 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276736 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276739 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276742 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276745 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.297579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276748 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276750 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276753 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276757 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276761 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276765 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276770 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276774 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276779 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276784 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276788 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276792 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276795 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276798 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276803 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276808 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.298068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.276839 4622 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276886 4622 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276891 4622 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276895 4622 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276897 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276900 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276902 4622 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276905 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276909 4622 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276913 4622 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276916 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276919 4622 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276923 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276928 4622 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276932 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276937 4622 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276942 4622 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276947 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276951 4622 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:12:52.298544 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276956 4622 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276961 4622 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276965 4622 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276970 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276974 4622 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276979 4622 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276984 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276988 4622 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276992 4622 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.276996 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277000 4622 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277004 4622 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277009 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277013 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277016 4622 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277019 4622 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277022 4622 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277024 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277027 4622 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:12:52.299084 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277030 4622 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277032 4622 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277054 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277058 4622 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277061 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277064 4622 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277067 4622 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277070 4622 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277073 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277075 4622 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277078 4622 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277081 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277084 4622 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277087 4622 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277090 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277092 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277095 4622 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277098 4622 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277103 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:12:52.299718 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277106 4622 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277109 4622 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277112 4622 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277114 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277117 4622 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277120 4622 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277123 4622 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277126 4622 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277129 4622 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277132 4622 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277134 4622 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277137 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277140 4622 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277142 4622 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277145 4622 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277148 4622 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277151 4622 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277154 4622 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277157 4622 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277159 4622 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:12:52.300310 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277162 4622 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277169 4622 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277175 4622 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277183 4622 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277186 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277189 4622 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277192 4622 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277195 4622 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277198 4622 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277200 4622 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277203 4622 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277208 4622 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277211 4622 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277214 4622 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277216 4622 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277219 4622 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277222 4622 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277225 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277227 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:12:52.301082 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277230 4622 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277233 4622 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277236 4622 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277239 4622 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277241 4622 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277244 4622 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277247 4622 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277251 4622 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277253 4622 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277279 4622 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:52.277283 4622 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.277381 4622 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.278837 4622 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.278901 4622 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.279126 4622 server.go:1014] "Starting client certificate rotation" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.279287 4622 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.279343 4622 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:07:08 +0000 UTC" deadline="2028-05-23 20:05:07.05857414 +0000 UTC" Oct 30 17:12:52.301829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.279375 4622 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13706h52m14.779201704s" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.279673 4622 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.280968 4622 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.282568 4622 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.284426 4622 log.go:25] "Validated CRI v1 image API" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.285249 4622 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:12:52.302493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.287099 4622 fs.go:135] Filesystem UUIDs: map[3896f177-1054-4d15-8f42-2651d1c0aa7a:/dev/nvme0n1p4 6dbb4951-afda-417f-9324-56e1a56584b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:12:52.302739 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.287124 4622 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fd390e02b7b804d0df8c7923cca93fc5a9734c92d0b5a1dc7611cb4d4671840/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fd390e02b7b804d0df8c7923cca93fc5a9734c92d0b5a1dc7611cb4d4671840/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7/userdata/shm major:0 minor:67 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26243dda-d8a0-457d-8844-81a32a6def00/volumes/kubernetes.io~projected/kube-api-access-7lgdl:{mountpoint:/var/lib/kubelet/pods/26243dda-d8a0-457d-8844-81a32a6def00/volumes/kubernetes.io~projected/kube-api-access-7lgdl major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~projected/kube-api-access-5t8fg:{mountpoint:/var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~projected/kube-api-access-5t8fg major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~secret/metrics-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a/volumes/kubernetes.io~projected/kube-api-access-htnkr:{mountpoint:/var/lib/kubelet/pods/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a/volumes/kubernetes.io~projected/kube-api-access-htnkr major:0 minor:87 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39f1686a-04e8-4f10-874a-51e855eea9a0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/39f1686a-04e8-4f10-874a-51e855eea9a0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40e85761-926d-4b83-9df4-e5be2b5844c4/volumes/kubernetes.io~projected/kube-api-access-f57bx:{mountpoint:/var/lib/kubelet/pods/40e85761-926d-4b83-9df4-e5be2b5844c4/volumes/kubernetes.io~projected/kube-api-access-f57bx major:0 minor:89 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:74 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~projected/kube-api-access-4j7h6:{mountpoint:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~projected/kube-api-access-4j7h6 major:0 minor:76 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~projected/kube-api-access-vllp8:{mountpoint:/var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~projected/kube-api-access-vllp8 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~projected/kube-api-access-v8s9t:{mountpoint:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~projected/kube-api-access-v8s9t major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~projected/kube-api-access-tzw7s:{mountpoint:/var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~projected/kube-api-access-tzw7s major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~secret/metrics-certs major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~projected/kube-api-access-fcc2d:{mountpoint:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~projected/kube-api-access-fcc2d major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c/volumes/kubernetes.io~secret/tls-certificates major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aadcc9ba-2535-48cd-abef-1535e15f6ec6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/aadcc9ba-2535-48cd-abef-1535e15f6ec6/volumes/kubernetes.io~secret/agent-certs major:0 minor:75 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~projected/kube-api-access-wxlxl:{mountpoint:/var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~projected/kube-api-access-wxlxl major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbc52026-7904-40fd-95ec-1c7b3db1b5cf/volumes/kubernetes.io~projected/kube-api-access-ww7qb:{mountpoint:/var/lib/kubelet/pods/bbc52026-7904-40fd-95ec-1c7b3db1b5cf/volumes/kubernetes.io~projected/kube-api-access-ww7qb major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d078773f-7b19-4a09-a258-a996048df5aa/volumes/kubernetes.io~projected/kube-api-access-hwdv9:{mountpoint:/var/lib/kubelet/pods/d078773f-7b19-4a09-a258-a996048df5aa/volumes/kubernetes.io~projected/kube-api-access-hwdv9 major:0 minor:102 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2f5736c-1133-4e5b-88b0-181bd447b670/volumes/kubernetes.io~projected/kube-api-access-xh2ff:{mountpoint:/var/lib/kubelet/pods/e2f5736c-1133-4e5b-88b0-181bd447b670/volumes/kubernetes.io~projected/kube-api-access-xh2ff major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed427dd8-842e-4b41-9436-5e02ed499871/volumes/kubernetes.io~projected/kube-api-access-wcpsx:{mountpoint:/var/lib/kubelet/pods/ed427dd8-842e-4b41-9436-5e02ed499871/volumes/kubernetes.io~projected/kube-api-access-wcpsx major:0 minor:88 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~projected/kube-api-access-g4xwb:{mountpoint:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~projected/kube-api-access-g4xwb major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:307 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/adc2d064f92861894e5fac3e745f5db6185c85d416327562da535072c37b282b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/ed6c1b1cdc5e2c805b02fd3af2e9f94cccbb36161934a86ebe27887c0ebe87c2/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/bb200a016dd742cbb7d0ad833cb7e2dea68b2e26e8a7b863c0bdc0923d9dd75c/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/83d6c1b7e04a0910739522a79a0debcc4c527235a6ea9de0f43127957fa90fcb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/966be0450267d2f2b0d47951bdd005875ddd4ca2738e123907f6977a358c8b6d/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7c023aac4b28a54bcdd3053be87e5c42080c68d4e4551036d4a8f73020a2db5b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5f13812350d78061d063252c0f21a60e95a9db4e1da020d8e17001d1a3091bd5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c41c04daf705e67edaa20cd813daad52a908fa22330e82db8e8e6b831faf7aff/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/db282a46a4135660d35dc981035b58b3739b30afbf7770ca17ff9683b9295e49/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/4ebe744a63bda1aa8fc75bc44ad22492c3931fc778d9f0c3d6fc1de2468c8e51/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/82b34c488f9b5dfba325b91f5d370ce0d308a82540a39ad018f97d26cf18855a/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6352e22c4b61030bd9cb0614b389a741826a4bc7abe5ae0a5783354317d29e05/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6be6aed8fdfd7077dce170d388a69a027224df9d1ac0c4816ed2c13cf32cb637/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ef2b173c589123455ed883a561d378d3d1d9e1674ee50dfd7e79ad6e80014102/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/772feddbc1e5d4a9921e5d91c68c5893c43fd7d909587a33e682db8d4f9242de/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/adc2d8223fe7a7ccb12ebd8af4901db56f63c6f35e7d75cc4c3ac3520759c618/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1fcffc16afae0864c8ab30131cba100e6de6031c8a9accfba05d57359db6b691/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/f51617cd0491b53b4cbf4e66f0086bea8c8a91a379273eeea6aac82b5aa512cb/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/ed9dd9d199853e6850521b0289fbfae23157b64fbf9c04eca0c9162b785a0a4a/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/ae4f4bf941f98fdefbad4a8ca32600b3e1d6459095a93a7858832d6977148ee4/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/4fa6dc92271a49a0e7be9749fd49bdf8d2d9cd3354eebc9404daee61252ed6b1/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/41b8c08da662b5f9a17041ef95ed487a9cae20389b8da057b0e592751bc057dc/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/75d75f7e514410bc79885790b3a5e215a1f880edf5d7b26f1f7f1eb60d9ccd2f/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/1c1a16f34d8d8b7902bfd5f7e109fc872c80797c695c303d6e526658ed5f7b87/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/3836916e9acb83fb365881d53635d747e19958b91891379f9f2f7669949558fe/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/43b1394e0a971f26aeda0015e5e3b57100d7333aea89dcd27208927eb3046df1/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/8c079de560513855bef24e29567ca6743188ec5198eb6ed168c5dca1a9ce15e2/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/73845d42ace1d4608e1739b4faeb466c448a6072df870006b18a11bb71a3364b/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c91a6d8e319272d1572b1b03febbb553d0fbecb6704e595d9f618e18e65785c2/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/746620883d1ba7888029f00e87503429769c494c14fe2325e2cccf0c000893a0/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/9e69c4754fd2a2170b89e50ad26a94214814410f3e6ad8c783bbf8f579b3ddc1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/9bdbc593486b3fb04705b71c825099cad3eea3bb546c7d15014319a37b9f673f/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/0bbec103a29b20fe9e5c158f16199958279b872d96dbdcec9c3e3373a45a5ad5/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/9fa2259129e7efd635d229fb4cfb40a6230f9540630d26ed0396ece60e64af66/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/1d6e331c61dec4ce9e45bb422a9434547b23c8370ecd59122134b2d163bebfc6/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/7c0a639d1c0a8f64f536254aa766ec39ab34d493e2889f33bf8c7add1573217b/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/b824fab1a05e377e7d6f7ccf9b60eb9f9b998e4cdcbee4a7458b87a641dad9cc/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/4de78592450d958b671a803c8515e4403164df6260a050d0874cc8135ec8fcde/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/a3865f30f6337967fa786d93f57bbfc9c6e976f8df5094825bbd64a8d77e46cd/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/394ba0e8c7b483c70321a0945f4fe6802fe06abcf32cf256a5b1d0c9c40d5c99/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/7c019dbc8fd1d8984d8a999738287813e461e38a7c8b6ae79506edaaf9b242c4/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/460ed3ce4206688c3f52b5cea303d94fb082fc56857c687e20ae5953d897a17c/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/49df5b055178087dbd05d0877363fc18874027135577ded58de833e539f61971/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/14b884f26b5e7cbb6fbe83ec98065731573f3836a5e1780c967e4a4b2a52d9fc/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/c6d2dc9745a0469200924fb4d8691161f8859829e72f03e89730d05b4870fd67/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-58:{mountpoint:/var/lib/containers/storage/overlay/6794b4a79300f19fadfaa3d5833745cae284c79c1eda2d245d55dfacced42cca/merged major:0 minor:58 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/507f39ad1e9eb5289e303fd65363d6e66b5c6ba8d3b20b5d5f481d9f1227585f/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/d63b5432826eb0501f5e0b1961acecae28ee43f655a095f7861bd15725e44f1a/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/1322a945180a03b67a127c18da92a75e98276d35c14e43afc87ea460b6052503/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/1a7351f7a7da32bcb6a53e0440e15af5a58bb6c2d10b093ad3dd51b42e1c1607/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4ab480aa7c77aefae9fb36a59f51fd34d63df0e637fd1780fb9211d6a26f4a7f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ca8257536177474b3ef740b2f9b876dbc23e6d35c41f871ef54c6d8f38d69192/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.296860 4622 manager.go:217] Machine: {Timestamp:2026-10-30 17:12:52.296119675 +0000 UTC m=+0.073591621 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099761 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec279fffb43a539a578993507e1c6932 SystemUUID:ec279fff-b43a-539a-5789-93507e1c6932 BootID:0da6a39f-24f0-4dbc-8ec9-feaa56bf8c3d Filesystems:[{Device:/run/containers/storage/overlay-containers/c7444c0609b3463a32d56523d75bd953ec5cbb2b6aa7d1f575522a8dd2d4b8b7/userdata/shm DeviceMajor:0 DeviceMinor:67 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed427dd8-842e-4b41-9436-5e02ed499871/volumes/kubernetes.io~projected/kube-api-access-wcpsx DeviceMajor:0 DeviceMinor:88 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/245b1545c6c41b12ce22a7710643e261ce4be5ce8e5b8aecbb8623dbc89d5a4c/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~projected/kube-api-access-v8s9t DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26243dda-d8a0-457d-8844-81a32a6def00/volumes/kubernetes.io~projected/kube-api-access-7lgdl DeviceMajor:0 DeviceMinor:70 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5af1f0aaff5abe987a507282c3a2790a734e83b2eb75936e87795ca3977d92b/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a/volumes/kubernetes.io~projected/kube-api-access-htnkr DeviceMajor:0 DeviceMinor:87 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/651a4deb3aafcd8a1e7f0aa4f2454a9500c0ab07480dc1c5e7aaa9c997375c7d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~projected/kube-api-access-vllp8 DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9edbcd0c0a1531c9cd333f9af0db51c685cdfb1df328b20cfae1c65f5fd0fa3c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2f5736c-1133-4e5b-88b0-181bd447b670/volumes/kubernetes.io~projected/kube-api-access-xh2ff DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbe6b0f78df9da9703ed8a15c173b7dbba488d23d8e20b52662a250880bef77d/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/volumes/kubernetes.io~projected/kube-api-access-g4xwb DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aadcc9ba-2535-48cd-abef-1535e15f6ec6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:75 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39f1686a-04e8-4f10-874a-51e855eea9a0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~projected/kube-api-access-tzw7s DeviceMajor:0 DeviceMinor:72 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/405a01c4788d62703de23ff15ef1843ef8c82789269c48cfcc37e181d4f7c1b7/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/965c6a5a28384a65b6b44d438aa2cafaee76388406db691e6a0020d24723883d/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:242 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:74 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~projected/kube-api-access-wxlxl DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3da71ff227014a24ad5518bf72402cdb85d0937121ccbe5302ccbeed75f69e35/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6f11b003c158fd2f0178298fb07fe6ea98d87507d7eb7205eb7ecfa9e8979a8/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~projected/kube-api-access-5t8fg DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40ebef61-196f-4e79-adf1-20bea663abc3/volumes/kubernetes.io~projected/kube-api-access-4j7h6 DeviceMajor:0 DeviceMinor:76 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d078773f-7b19-4a09-a258-a996048df5aa/volumes/kubernetes.io~projected/kube-api-access-hwdv9 DeviceMajor:0 DeviceMinor:102 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/affacaf7-208e-4092-9aaf-3adba31e8934/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6d89b5f3-c0cb-4878-9496-768483a56be9/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee63db4-03fa-415b-9489-b188e6a71575/volumes/kubernetes.io~projected/kube-api-access-fcc2d DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c23856e-00f0-409e-a47b-6754355fd5ce/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fd390e02b7b804d0df8c7923cca93fc5a9734c92d0b5a1dc7611cb4d4671840/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40e85761-926d-4b83-9df4-e5be2b5844c4/volumes/kubernetes.io~projected/kube-api-access-f57bx DeviceMajor:0 DeviceMinor:89 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc8e8429416523054d4eb62e8a5f55a70aceae97ae15d82203cfa525ab78f5b3/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6603e9dd0f63058342b26a4ef08959a6d1494404899e46d1fd53973a1e6deb6/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:196 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbc52026-7904-40fd-95ec-1c7b3db1b5cf/volumes/kubernetes.io~projected/kube-api-access-ww7qb DeviceMajor:0 DeviceMinor:71 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-58 DeviceMajor:0 DeviceMinor:58 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ecdb3a8a55344e5e9357a9e4ea125492f21af25a894f25126953e338329c827/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/503e502918d296ea02440f5171ffc2a72e4dc22ed4adb3b17598932a2701d3c4/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/320a0cd8-4ba1-4247-b0c7-3cfc8876b471/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/812f3cc0e9b9d91ebeeae5cec129f0e0ba0b3c735ae5c94f6322e233fb9d932d/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3da71ff227014a2 MacAddress:96:bc:b2:d4:b5:d4 Speed:10000 Mtu:8901} {Name:4fd390e02b7b804 MacAddress:8e:ba:db:f0:ad:f8 Speed:10000 Mtu:8901} {Name:812f3cc0e9b9d91 MacAddress:e2:91:70:2b:e0:4d Speed:10000 Mtu:8901} {Name:965c6a5a28384a6 MacAddress:46:83:76:f5:16:3b Speed:10000 Mtu:8901} {Name:b6f11b003c158fd MacAddress:52:9d:a1:1a:62:22 Speed:10000 Mtu:8901} {Name:bc8e84294165230 MacAddress:92:fa:52:8e:c2:b1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:8d:a1:93 Speed:0 Mtu:9001} {Name:br-int MacAddress:de:a4:a0:f1:61:9e Speed:0 Mtu:8901} {Name:c8dd9b77ea96aae MacAddress:5e:75:ae:3e:fd:a1 Speed:10000 Mtu:8901} {Name:d6603e9dd0f6305 MacAddress:7e:2c:89:32:bf:f1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:8d:a1:93 Speed:0 Mtu:9001} {Name:fbe6b0f78df9da9 MacAddress:d6:54:6e:52:61:24 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:12:90:76:4a:e6:45 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:2b:b2:af:f3:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297209 4622 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297323 4622 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297636 4622 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297661 4622 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297806 4622 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297814 4622 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297829 4622 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297849 4622 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.297935 4622 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298102 4622 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298206 4622 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298240 4622 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298253 4622 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298283 4622 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298292 4622 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298302 4622 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298929 4622 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.298940 4622 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:12:52.303060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.300093 4622 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:12:52.319360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319329 4622 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:12:52.319360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319348 4622 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:12:52.319580 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319490 4622 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:12:52.319755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319739 4622 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:12:52.319817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319782 4622 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:12:52.319930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319912 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:12:52.319930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319930 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:12:52.319930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319937 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319943 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319948 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319954 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319960 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319967 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319976 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.319984 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.320002 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.320012 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.320048 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:12:52.320118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.320054 4622 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:12:52.323720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.323703 4622 server.go:1263] "Started kubelet" Oct 30 17:12:52.323835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.323777 4622 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:12:52.323961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.323913 4622 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:12:52.324005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.323984 4622 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:12:52.324773 ip-10-0-143-147 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:12:52.326413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.324782 4622 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:12:52.326413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.325298 4622 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:12:52.330477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.329857 4622 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:12:52.330477 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.330069 4622 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-147.ec2.internal\" not found" Oct 30 17:12:52.330477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.330124 4622 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:12:52.330477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.330138 4622 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:12:52.330477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.330141 4622 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:12:52.331010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.330976 4622 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:12:52.331412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.331388 4622 factory.go:55] Registering systemd factory Oct 30 17:12:52.331412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.331413 4622 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:12:52.332892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.332351 4622 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:07:36 +0000 UTC" deadline="2028-05-21 08:47:12.730231808 +0000 UTC" Oct 30 17:12:52.332892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.332396 4622 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13647h34m20.397839987s" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.332966 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" volumeName="kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333001 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320a0cd8-4ba1-4247-b0c7-3cfc8876b471" volumeName="kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333014 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a" volumeName="kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333026 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40e85761-926d-4b83-9df4-e5be2b5844c4" volumeName="kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333039 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c23856e-00f0-409e-a47b-6754355fd5ce" volumeName="kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333049 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333059 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d078773f-7b19-4a09-a258-a996048df5aa" volumeName="kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333071 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2f5736c-1133-4e5b-88b0-181bd447b670" volumeName="kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333085 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320a0cd8-4ba1-4247-b0c7-3cfc8876b471" volumeName="kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333095 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c23856e-00f0-409e-a47b-6754355fd5ce" volumeName="kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333106 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c23856e-00f0-409e-a47b-6754355fd5ce" volumeName="kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333117 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333128 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333141 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333155 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee63db4-03fa-415b-9489-b188e6a71575" volumeName="kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333167 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee63db4-03fa-415b-9489-b188e6a71575" volumeName="kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333180 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320a0cd8-4ba1-4247-b0c7-3cfc8876b471" volumeName="kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333192 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ebef61-196f-4e79-adf1-20bea663abc3" volumeName="kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333203 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aadcc9ba-2535-48cd-abef-1535e15f6ec6" volumeName="kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333214 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="affacaf7-208e-4092-9aaf-3adba31e8934" volumeName="kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333225 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" volumeName="kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333237 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b" volumeName="kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333250 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b" volumeName="kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333280 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b" volumeName="kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333293 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26243dda-d8a0-457d-8844-81a32a6def00" volumeName="kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333303 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a" volumeName="kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333319 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a" volumeName="kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333341 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ebef61-196f-4e79-adf1-20bea663abc3" volumeName="kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333373 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="909cd2de-faa4-4b8f-a0b1-5bc4e261b32c" volumeName="kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333393 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="affacaf7-208e-4092-9aaf-3adba31e8934" volumeName="kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333414 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed427dd8-842e-4b41-9436-5e02ed499871" volumeName="kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333438 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="320a0cd8-4ba1-4247-b0c7-3cfc8876b471" volumeName="kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333459 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333474 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee63db4-03fa-415b-9489-b188e6a71575" volumeName="kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333486 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d078773f-7b19-4a09-a258-a996048df5aa" volumeName="kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333497 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed427dd8-842e-4b41-9436-5e02ed499871" volumeName="kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333508 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aadcc9ba-2535-48cd-abef-1535e15f6ec6" volumeName="kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333519 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b" volumeName="kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333531 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c23856e-00f0-409e-a47b-6754355fd5ce" volumeName="kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333543 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d89b5f3-c0cb-4878-9496-768483a56be9" volumeName="kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333555 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d89b5f3-c0cb-4878-9496-768483a56be9" volumeName="kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333567 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee63db4-03fa-415b-9489-b188e6a71575" volumeName="kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333579 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee63db4-03fa-415b-9489-b188e6a71575" volumeName="kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333593 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" volumeName="kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333605 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" volumeName="kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333617 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333630 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26243dda-d8a0-457d-8844-81a32a6def00" volumeName="kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333642 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39f1686a-04e8-4f10-874a-51e855eea9a0" volumeName="kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333653 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40ebef61-196f-4e79-adf1-20bea663abc3" volumeName="kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333665 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333676 4622 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6723eff8-01c0-484c-88eb-7f363b0e565a" volumeName="kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert" seLinuxMountContext="" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333688 4622 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333696 4622 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333711 4622 factory.go:153] Registering CRI-O factory Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333729 4622 factory.go:223] Registration of the crio container factory successfully Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333787 4622 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333810 4622 factory.go:103] Registering Raw factory Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.333827 4622 manager.go:1199] Started watching for new ooms in manager Oct 30 17:12:52.334154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.334193 4622 manager.go:319] Starting recovery of all containers Oct 30 17:12:52.358772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.358574 4622 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:52.359053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.359032 4622 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:12:52.364452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.364421 4622 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:12:52.366139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.366118 4622 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:12:52.366139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.366140 4622 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:12:52.366329 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.366160 4622 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:12:52.366329 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.366202 4622 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:12:52.368943 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.368924 4622 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:52.370186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.370164 4622 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:52.379352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.379334 4622 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:12:52.407974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407938 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="58f1644d7710daa5591e129e33a66212f4fe9e8e3691964caaee201eaeda39aa" exitCode=0 Oct 30 17:12:52.407974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407967 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="7cab40f5fbb6d1a0b36e3420b3f4b13d0983c9628c219cff8a818cbac9f7b973" exitCode=0 Oct 30 17:12:52.407974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407976 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="0c06b6380aa4cbd9f2fdb5ad92f95fa74fffe10c8d7ade4ecfa5214766ce5974" exitCode=0 Oct 30 17:12:52.407974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407985 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="bd6a95c9fc3243897a09803a8131a0bc57ad237b00a170b0d986e81d05f31c56" exitCode=0 Oct 30 17:12:52.408316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407993 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="0f002132a6b5e1f820a9a06703144c0de123bcdbc07dea603606d7e5edcb4e07" exitCode=0 Oct 30 17:12:52.408316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.407999 4622 generic.go:356] "Generic (PLEG): container finished" podID="bbc52026-7904-40fd-95ec-1c7b3db1b5cf" containerID="359574c158fad49c3369e8a87311c125fd066b3e8d22c4edfb3f00bc7d28fead" exitCode=0 Oct 30 17:12:52.429708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.429613 4622 generic.go:356] "Generic (PLEG): container finished" podID="01c50f6fe0fd9f68f1ab538d873faa96" containerID="2b1fc76f4ba5a972e7e737894e118a25856776b3c719c027b37f9e5732b7acc3" exitCode=0 Oct 30 17:12:52.430946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.430926 4622 manager.go:324] Recovery completed Oct 30 17:12:52.450989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.450960 4622 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:12:52.450989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.450977 4622 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:12:52.450989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.450997 4622 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451123 4622 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451134 4622 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451162 4622 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451168 4622 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451173 4622 policy_none.go:50] "Start" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451180 4622 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:12:52.451197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451189 4622 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:12:52.451411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451329 4622 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:12:52.451411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451337 4622 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:12:52.451411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.451342 4622 policy_none.go:44] "Start" Oct 30 17:12:52.456292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.455767 4622 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:12:52.461120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461106 4622 manager.go:341] "Starting Device Plugin manager" Oct 30 17:12:52.461277 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.461250 4622 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:12:52.461277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461276 4622 server.go:87] "Starting device plugin registration server" Oct 30 17:12:52.461593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461583 4622 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:12:52.461625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461596 4622 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:12:52.461723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461704 4622 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:12:52.461849 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461810 4622 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:12:52.461966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461856 4622 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:12:52.461966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461863 4622 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:12:52.462040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.461923 4622 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:12:52.461844065Z","UUID":"8c7574de-e347-40c8-a994-0a8f993a274c","Handler":null,"Name":"","Endpoint":""} Oct 30 17:12:52.463169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.463146 4622 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:12:52.463253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.463189 4622 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:12:52.463669 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.463651 4622 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:12:52.466296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.466247 4622 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal"] Oct 30 17:12:52.466887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.466872 4622 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.466971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.466931 4622 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.467037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.467007 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"a33e219052cd1905654f6e2cbcaae521","Type":"ContainerStarted","Data":"ecc98d87375ffdda340fa2e6b6f44b2acca0c1736bc658148eb316e062735c09"} Oct 30 17:12:52.467086 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.467056 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" event={"ID":"a33e219052cd1905654f6e2cbcaae521","Type":"ContainerStarted","Data":"f5ef939874d61795be94d7286dae86ae07cedb0fdc7dbb8b3b6259168ade3488"} Oct 30 17:12:52.467122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.467095 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"2a68b8bf2a646a982a556da4356f3660b4aeca39aa52a4cd2f50bbdc5d0c571f"} Oct 30 17:12:52.467122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.467110 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerDied","Data":"2b1fc76f4ba5a972e7e737894e118a25856776b3c719c027b37f9e5732b7acc3"} Oct 30 17:12:52.467185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.467124 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" event={"ID":"01c50f6fe0fd9f68f1ab538d873faa96","Type":"ContainerStarted","Data":"faf12f33440d0334ed3fa8c5d053ddb4fc9c864431d468aa2acf37131e257521"} Oct 30 17:12:52.490046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.490013 4622 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:52.490203 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.490071 4622 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.492790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.492773 4622 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:52.492891 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:52.492799 4622 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.560245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.560209 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.560245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.560245 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.560463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.560291 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.562316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.562292 4622 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:12:52.563633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.563613 4622 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:12:52.563720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.563643 4622 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:12:52.563720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.563652 4622 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-147.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:12:52.563720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.563686 4622 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.594204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.594173 4622 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.594375 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.594241 4622 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660430 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660466 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660525 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660584 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/01c50f6fe0fd9f68f1ab538d873faa96-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" (UID: \"01c50f6fe0fd9f68f1ab538d873faa96\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660607 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:52.660684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:52.660633 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a33e219052cd1905654f6e2cbcaae521-config\") pod \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" (UID: \"a33e219052cd1905654f6e2cbcaae521\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:53.298532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.298498 4622 apiserver.go:52] "Watching apiserver" Oct 30 17:12:53.333541 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.333511 4622 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:12:53.333986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.333968 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lr6fs","kube-system/global-pull-secret-syncer-854ml","kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal","openshift-ingress-canary/ingress-canary-r9628","openshift-multus/network-metrics-daemon-lqgx2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv","openshift-network-operator/iptables-alerter-w4gnb","openshift-cluster-node-tuning-operator/tuned-2k8mk","openshift-dns/dns-default-98bcg","openshift-dns/node-resolver-m72vs","openshift-insights/insights-runtime-extractor-tzxkz","openshift-monitoring/prometheus-operator-9cd89b74c-svgft","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal","openshift-network-diagnostics/network-check-target-wlnpk","openshift-image-registry/node-ca-bgdc6","openshift-multus/multus-77wmf","kube-system/konnectivity-agent-rzvbg","openshift-console/console-58cd6766d5-btnxw","openshift-ovn-kubernetes/ovnkube-node-gb9j4"] Oct 30 17:12:53.348010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.347974 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.348847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.348818 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-n92rc\"" Oct 30 17:12:53.348847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.348842 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:12:53.349064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.348948 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.349284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.349247 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ctbv7\"" Oct 30 17:12:53.349421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.349252 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:12:53.350122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.350101 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:12:53.355396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.355376 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:12:53.355595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.355580 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.358141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.358128 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.358737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.358724 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:12:53.359340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359329 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.359568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359557 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:12:53.359708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359696 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.359866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359848 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.359958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359877 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.359958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359880 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.359958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359899 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.359958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.359904 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.360308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360288 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mkzvv\"" Oct 30 17:12:53.360390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360317 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:12:53.360390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360312 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-49f6v\"" Oct 30 17:12:53.360472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360398 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:12:53.360522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360501 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 17:12:53.360522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360504 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zkp55\"" Oct 30 17:12:53.360587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360521 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wklnr\"" Oct 30 17:12:53.360587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360545 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pnbwd\"" Oct 30 17:12:53.360674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360547 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:12:53.360674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360594 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:12:53.360756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360679 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2wbm5\"" Oct 30 17:12:53.360756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.360713 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8szd\"" Oct 30 17:12:53.363778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.363766 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lrx5\"" Oct 30 17:12:53.364121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.364109 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:12:53.364256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.364244 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 17:12:53.365151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365135 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jlm27\"" Oct 30 17:12:53.365151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365145 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6mz24\"" Oct 30 17:12:53.365335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365319 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:12:53.365411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365396 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vxcg\"" Oct 30 17:12:53.365682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365669 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn7sb\"" Oct 30 17:12:53.365761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365748 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vqnr7\"" Oct 30 17:12:53.365809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365773 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:12:53.365809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365784 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rz5ks\"" Oct 30 17:12:53.365809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365802 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:12:53.365916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365811 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g55vh\"" Oct 30 17:12:53.365916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365819 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5c2r9\"" Oct 30 17:12:53.365916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365836 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 17:12:53.365916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.365838 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:12:53.367711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367698 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:12:53.367829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367810 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.367901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367832 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:12:53.367901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367855 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.367901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367860 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:12:53.367901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.367867 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.368684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.368668 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.368684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.368668 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.368819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.368753 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:12:53.368819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.368809 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:12:53.368900 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.368850 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.370093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370078 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.370093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370086 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370088 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370110 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370126 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370128 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370149 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370163 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.370213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370197 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:12:53.370461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370336 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:12:53.370461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370357 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:12:53.370461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370363 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:12:53.370461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.370377 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:12:53.374296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.374281 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:12:53.433186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.433162 4622 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:12:53.434470 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.434449 4622 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:53.434597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.434576 4622 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:53.455702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.455677 4622 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:53.455702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.455687 4622 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:12:53.455876 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:53.455732 4622 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-147.ec2.internal" Oct 30 17:12:53.455876 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:53.455739 4622 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-147.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-147.ec2.internal" Oct 30 17:12:53.464709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464675 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.464841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464729 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.464841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464754 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.464841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464770 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xwb\" (UniqueName: \"kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.464841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464791 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.465094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464863 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.465094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.464999 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.465408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465387 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcc2d\" (UniqueName: \"kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465422 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465135 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465451 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465134 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465134 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.465486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465475 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465520 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465585 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bk4wv\" (UID: \"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465616 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7qb\" (UniqueName: \"kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465647 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465664 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-etc-tuned\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.465771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465674 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465775 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465800 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465845 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ed427dd8-842e-4b41-9436-5e02ed499871-iptables-alerter-script\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465866 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465915 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/909cd2de-faa4-4b8f-a0b1-5bc4e261b32c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bk4wv\" (UID: \"909cd2de-faa4-4b8f-a0b1-5bc4e261b32c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465919 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465962 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.465989 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466016 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466043 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466043 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466054 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466066 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466105 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466133 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466158 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466187 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466233 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466367 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-config\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466395 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466415 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cni-binary-copy\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466431 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466419 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-daemon-config\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466431 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ee63db4-03fa-415b-9489-b188e6a71575-ovn-node-metrics-cert\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466472 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466492 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466510 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466508 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aadcc9ba-2535-48cd-abef-1535e15f6ec6-konnectivity-ca\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466582 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466630 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26243dda-d8a0-457d-8844-81a32a6def00-tmp-dir\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466642 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466676 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466713 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466789 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-env-overrides\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466800 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.466891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466878 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466908 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466922 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ee63db4-03fa-415b-9489-b188e6a71575-ovnkube-script-lib\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466925 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.466955 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxlxl\" (UniqueName: \"kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467013 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467034 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-tmp-dir\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467107 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllp8\" (UniqueName: \"kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467149 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467171 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467334 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467334 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467367 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467386 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7h6\" (UniqueName: \"kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.467478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467473 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467496 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467514 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgdl\" (UniqueName: \"kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467529 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467566 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467601 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467643 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467670 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467702 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467720 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467729 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467779 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdv9\" (UniqueName: \"kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467827 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpsx\" (UniqueName: \"kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467835 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39f1686a-04e8-4f10-874a-51e855eea9a0-original-pull-secret\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467905 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.467996 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57bx\" (UniqueName: \"kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468014 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468077 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:53.468109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468105 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468131 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468154 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468186 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468215 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468254 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468309 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468339 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-metrics-tls\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468358 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468360 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aadcc9ba-2535-48cd-abef-1535e15f6ec6-agent-certs\") pod \"konnectivity-agent-rzvbg\" (UID: \"aadcc9ba-2535-48cd-abef-1535e15f6ec6\") " pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468424 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468431 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468471 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468559 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468564 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c23856e-00f0-409e-a47b-6754355fd5ce-data-volume\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468611 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468633 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468666 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468704 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.468811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.468738 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469087 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/affacaf7-208e-4092-9aaf-3adba31e8934-cert\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469109 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469223 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c23856e-00f0-409e-a47b-6754355fd5ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469242 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnkr\" (UniqueName: \"kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469336 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469417 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469454 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.469526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469521 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469557 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469600 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469779 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469855 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469884 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.469923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469883 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.469967 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470004 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470024 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d078773f-7b19-4a09-a258-a996048df5aa-serviceca\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470033 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470069 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470102 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470141 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8fg\" (UniqueName: \"kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.470199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470180 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470214 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470247 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470246 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470354 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470393 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470423 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470460 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw7s\" (UniqueName: \"kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:53.470582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470495 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.470944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.470773 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-config-volume\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.472583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.471105 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40ebef61-196f-4e79-adf1-20bea663abc3-tmp\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.472583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.471348 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.472583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.471583 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d89b5f3-c0cb-4878-9496-768483a56be9-metrics-certs\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:53.485563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.485537 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcc2d\" (UniqueName: \"kubernetes.io/projected/6ee63db4-03fa-415b-9489-b188e6a71575-kube-api-access-fcc2d\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.485659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.485641 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"console-58cd6766d5-btnxw\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:53.485714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.485661 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xwb\" (UniqueName: \"kubernetes.io/projected/f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b-kube-api-access-g4xwb\") pod \"prometheus-operator-9cd89b74c-svgft\" (UID: \"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" Oct 30 17:12:53.486072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.486053 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7qb\" (UniqueName: \"kubernetes.io/projected/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-kube-api-access-ww7qb\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.486444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.486426 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2ff\" (UniqueName: \"kubernetes.io/projected/e2f5736c-1133-4e5b-88b0-181bd447b670-kube-api-access-xh2ff\") pod \"network-check-target-wlnpk\" (UID: \"e2f5736c-1133-4e5b-88b0-181bd447b670\") " pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:53.491514 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491493 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdv9\" (UniqueName: \"kubernetes.io/projected/d078773f-7b19-4a09-a258-a996048df5aa-kube-api-access-hwdv9\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.491702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491685 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7h6\" (UniqueName: \"kubernetes.io/projected/40ebef61-196f-4e79-adf1-20bea663abc3-kube-api-access-4j7h6\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.491927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491906 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxlxl\" (UniqueName: \"kubernetes.io/projected/affacaf7-208e-4092-9aaf-3adba31e8934-kube-api-access-wxlxl\") pod \"ingress-canary-r9628\" (UID: \"affacaf7-208e-4092-9aaf-3adba31e8934\") " pod="openshift-ingress-canary/ingress-canary-r9628" Oct 30 17:12:53.491960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491911 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllp8\" (UniqueName: \"kubernetes.io/projected/4c23856e-00f0-409e-a47b-6754355fd5ce-kube-api-access-vllp8\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.491990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491966 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57bx\" (UniqueName: \"kubernetes.io/projected/40e85761-926d-4b83-9df4-e5be2b5844c4-kube-api-access-f57bx\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.491990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.491972 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgdl\" (UniqueName: \"kubernetes.io/projected/26243dda-d8a0-457d-8844-81a32a6def00-kube-api-access-7lgdl\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.495809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.495793 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnkr\" (UniqueName: \"kubernetes.io/projected/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-kube-api-access-htnkr\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.498495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.498475 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8fg\" (UniqueName: \"kubernetes.io/projected/320a0cd8-4ba1-4247-b0c7-3cfc8876b471-kube-api-access-5t8fg\") pod \"dns-default-98bcg\" (UID: \"320a0cd8-4ba1-4247-b0c7-3cfc8876b471\") " pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:53.498570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.498550 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpsx\" (UniqueName: \"kubernetes.io/projected/ed427dd8-842e-4b41-9436-5e02ed499871-kube-api-access-wcpsx\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.498662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.498649 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw7s\" (UniqueName: \"kubernetes.io/projected/6d89b5f3-c0cb-4878-9496-768483a56be9-kube-api-access-tzw7s\") pod \"network-metrics-daemon-lqgx2\" (UID: \"6d89b5f3-c0cb-4878-9496-768483a56be9\") " pod="openshift-multus/network-metrics-daemon-lqgx2" Oct 30 17:12:53.571682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571602 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.571682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571633 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571651 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571666 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571681 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571698 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571718 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571719 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-netd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571719 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed427dd8-842e-4b41-9436-5e02ed499871-host-slash\") pod \"iptables-alerter-w4gnb\" (UID: \"ed427dd8-842e-4b41-9436-5e02ed499871\") " pod="openshift-network-operator/iptables-alerter-w4gnb" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571744 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-hostroot\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571752 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-conf-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571775 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571756 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571780 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-var-lib-kubelet\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571800 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26243dda-d8a0-457d-8844-81a32a6def00-hosts-file\") pod \"node-resolver-m72vs\" (UID: \"26243dda-d8a0-457d-8844-81a32a6def00\") " pod="openshift-dns/node-resolver-m72vs" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571804 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571825 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571840 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571865 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571866 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-log-socket\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571863 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-systemd\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571888 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-kubernetes\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571895 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571904 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-socket-dir-parent\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571920 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571936 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571962 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.571981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.571909 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-etc-kubernetes\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572009 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-system-cni-dir\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572011 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572027 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-run\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572043 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bbc52026-7904-40fd-95ec-1c7b3db1b5cf-cnibin\") pod \"multus-additional-cni-plugins-lr6fs\" (UID: \"bbc52026-7904-40fd-95ec-1c7b3db1b5cf\") " pod="openshift-multus/multus-additional-cni-plugins-lr6fs" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572015 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572047 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572086 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572093 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-kubelet-config\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572103 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572086 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-conf\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572149 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39f1686a-04e8-4f10-874a-51e855eea9a0-dbus\") pod \"global-pull-secret-syncer-854ml\" (UID: \"39f1686a-04e8-4f10-874a-51e855eea9a0\") " pod="kube-system/global-pull-secret-syncer-854ml" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572149 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572171 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-device-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572191 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572210 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-var-lib-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572219 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572236 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572243 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572247 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-sys-fs\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572299 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572325 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572333 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-sys\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572347 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572364 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-system-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572372 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572378 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c23856e-00f0-409e-a47b-6754355fd5ce-crio-socket\") pod \"insights-runtime-extractor-tzxkz\" (UID: \"4c23856e-00f0-409e-a47b-6754355fd5ce\") " pod="openshift-insights/insights-runtime-extractor-tzxkz" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572399 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572411 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-modprobe-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572426 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572450 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572451 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-multus-certs\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572453 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-etc-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572476 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-systemd\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572481 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysconfig\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572472 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572504 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572507 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-openvswitch\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572542 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572558 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572572 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572571 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-k8s-cni-cncf-io\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572587 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572622 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-bin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572613 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d078773f-7b19-4a09-a258-a996048df5aa-host\") pod \"node-ca-bgdc6\" (UID: \"d078773f-7b19-4a09-a258-a996048df5aa\") " pod="openshift-image-registry/node-ca-bgdc6" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572645 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572660 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572669 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-run-ovn\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572675 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572680 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-etc-sysctl-d\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572693 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572715 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-netns\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572729 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-cnibin\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572728 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-node-log\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572744 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572777 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572778 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-lib-modules\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572797 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572823 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572826 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-cni-multus\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572825 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-run-netns\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572846 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572852 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-socket-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572865 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-registration-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572870 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572877 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-slash\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572899 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572903 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-host-var-lib-kubelet\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572936 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572944 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40ebef61-196f-4e79-adf1-20bea663abc3-host\") pod \"tuned-2k8mk\" (UID: \"40ebef61-196f-4e79-adf1-20bea663abc3\") " pod="openshift-cluster-node-tuning-operator/tuned-2k8mk" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572954 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-kubelet\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572960 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572980 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.572996 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.572963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573009 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573012 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573050 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573055 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a-multus-cni-dir\") pod \"multus-77wmf\" (UID: \"37eb0a41-d5a7-4c71-ac24-e7e6a8bfda3a\") " pod="openshift-multus/multus-77wmf" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573050 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-host-cni-bin\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573077 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/40e85761-926d-4b83-9df4-e5be2b5844c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8ghl\" (UID: \"40e85761-926d-4b83-9df4-e5be2b5844c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8ghl" Oct 30 17:12:53.575143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.573081 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ee63db4-03fa-415b-9489-b188e6a71575-systemd-units\") pod \"ovnkube-node-gb9j4\" (UID: \"6ee63db4-03fa-415b-9489-b188e6a71575\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:53.642860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.642741 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:12:53.926860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:53.926781 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:12:54.195306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:54.195217 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:54.198670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:54.198645 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bk4wv" Oct 30 17:12:54.286480 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:54.286454 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:54.439915 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:54.439865 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" event={"ID":"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b","Type":"ContainerStarted","Data":"ec664ba343173f9d265aebd95a2f426f4849e6d42d4610c2f99059436876afb8"} Oct 30 17:12:54.440404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:54.439931 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-svgft" event={"ID":"f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b","Type":"ContainerStarted","Data":"048bbede2bfe32df778b71f903899af26056a6bd4b348ecdfda8f7d2ac30956d"} Oct 30 17:12:54.558997 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:12:54.558963 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:12:55.044360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:55.044334 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:12:55.394528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:55.394433 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:55.604873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:55.604836 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:12:57.011853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.011817 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gtwmg"] Oct 30 17:12:57.014440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.014418 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.063846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.063809 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:12:57.063989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.063908 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:12:57.063989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.063931 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:12:57.064057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.063999 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-726zt\"" Oct 30 17:12:57.099253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099217 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099280 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-tls\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099374 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-root\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099445 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099478 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-metrics-client-ca\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099501 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-sys\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099559 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-wtmp\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099627 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-textfile\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.099711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.099651 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52599\" (UniqueName: \"kubernetes.io/projected/500e754b-8401-4604-8d32-5105317e9dc6-kube-api-access-52599\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200527 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-tls\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200568 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-root\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200618 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200648 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-metrics-client-ca\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200712 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-root\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200735 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-sys\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200780 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-sys\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200782 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-wtmp\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200820 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-textfile\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.200865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200846 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52599\" (UniqueName: \"kubernetes.io/projected/500e754b-8401-4604-8d32-5105317e9dc6-kube-api-access-52599\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.201258 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200875 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.201258 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.200900 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-wtmp\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.201258 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.201071 4622 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:12:57.201258 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.201175 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-textfile\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.201457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.201376 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-metrics-client-ca\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.201457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.201384 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.204448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.204433 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-tls\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.204491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.204450 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/500e754b-8401-4604-8d32-5105317e9dc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.224711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.224683 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52599\" (UniqueName: \"kubernetes.io/projected/500e754b-8401-4604-8d32-5105317e9dc6-kube-api-access-52599\") pod \"node-exporter-gtwmg\" (UID: \"500e754b-8401-4604-8d32-5105317e9dc6\") " pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.327187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.327149 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gtwmg" Oct 30 17:12:57.337189 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:57.337148 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500e754b_8401_4604_8d32_5105317e9dc6.slice/crio-24addc747df318d6fdd31eeb18ca04fbcf4e2989f52141338dd7699387300fc5 WatchSource:0}: Error finding container 24addc747df318d6fdd31eeb18ca04fbcf4e2989f52141338dd7699387300fc5: Status 404 returned error can't find the container with id 24addc747df318d6fdd31eeb18ca04fbcf4e2989f52141338dd7699387300fc5 Oct 30 17:12:57.450844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:57.450807 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtwmg" event={"ID":"500e754b-8401-4604-8d32-5105317e9dc6","Type":"ContainerStarted","Data":"24addc747df318d6fdd31eeb18ca04fbcf4e2989f52141338dd7699387300fc5"} Oct 30 17:12:58.396978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.396950 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98bcg" Oct 30 17:12:58.429490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.429459 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:12:58.434296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.432749 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.453556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.453527 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:12:58.457285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457244 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:12:58.457408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457392 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:12:58.457444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457434 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:12:58.457474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457455 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:12:58.457582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457567 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:12:58.457645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.457618 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:12:58.458155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.458138 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kfr4n\"" Oct 30 17:12:58.461916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.461888 4622 generic.go:356] "Generic (PLEG): container finished" podID="500e754b-8401-4604-8d32-5105317e9dc6" containerID="727344bcf48c98da775784b96c618a4524de8192b1d0ce50fd61951384e5b97a" exitCode=0 Oct 30 17:12:58.462031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.461933 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtwmg" event={"ID":"500e754b-8401-4604-8d32-5105317e9dc6","Type":"ContainerDied","Data":"727344bcf48c98da775784b96c618a4524de8192b1d0ce50fd61951384e5b97a"} Oct 30 17:12:58.467459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.467411 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:12:58.481037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.478041 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:12:58.481037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.479283 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:12:58.510003 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.509972 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510041 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510068 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510123 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27fbj\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-kube-api-access-27fbj\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510149 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510178 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510333 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510381 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-config-out\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510420 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510449 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.510506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.510492 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.511058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.511020 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-web-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.511697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.511133 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621003 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621040 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621058 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621078 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-web-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621096 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621114 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621135 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621151 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621172 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27fbj\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-kube-api-access-27fbj\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621187 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621203 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621233 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621250 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-config-out\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.621862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.621794 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.622580 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.622554 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.623354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.622876 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef900d92-5256-4606-8198-ee3bfb001552-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.625460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.625411 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef900d92-5256-4606-8198-ee3bfb001552-config-out\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.625736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.625701 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-config-volume\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.625925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.625898 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.626188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.626152 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.626862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.626841 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.626986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.626964 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.627657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.627637 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.628668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.628647 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef900d92-5256-4606-8198-ee3bfb001552-web-config\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.629690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.629666 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.642529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.642502 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27fbj\" (UniqueName: \"kubernetes.io/projected/ef900d92-5256-4606-8198-ee3bfb001552-kube-api-access-27fbj\") pod \"alertmanager-main-0\" (UID: \"ef900d92-5256-4606-8198-ee3bfb001552\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:58.759953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:58.759865 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:12:59.100696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.100641 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:12:59.105116 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:59.105083 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef900d92_5256_4606_8198_ee3bfb001552.slice/crio-85a33ec817e5cbd4fedb978836ecf9ea43ea899fb15241401a99c93af4908eda WatchSource:0}: Error finding container 85a33ec817e5cbd4fedb978836ecf9ea43ea899fb15241401a99c93af4908eda: Status 404 returned error can't find the container with id 85a33ec817e5cbd4fedb978836ecf9ea43ea899fb15241401a99c93af4908eda Oct 30 17:12:59.239649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.239621 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78dc685f45-bspxf"] Oct 30 17:12:59.243556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.243534 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.257162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257132 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257165 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257180 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257221 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6k67353nslghe\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257241 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257254 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 17:12:59.257382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.257302 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x7xrs\"" Oct 30 17:12:59.268696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.268672 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78dc685f45-bspxf"] Oct 30 17:12:59.327275 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327241 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327301 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c735f137-8697-422b-b3e6-95852c822b4e-metrics-client-ca\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327367 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-grpc-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327394 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735f137-8697-422b-b3e6-95852c822b4e-tmp\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327429 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v6mj\" (UniqueName: \"kubernetes.io/projected/c735f137-8697-422b-b3e6-95852c822b4e-kube-api-access-9v6mj\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327478 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327550 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327599 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.327688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.327673 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.428614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428532 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.428614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428575 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.428614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428604 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c735f137-8697-422b-b3e6-95852c822b4e-metrics-client-ca\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428627 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-grpc-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428646 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735f137-8697-422b-b3e6-95852c822b4e-tmp\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428677 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v6mj\" (UniqueName: \"kubernetes.io/projected/c735f137-8697-422b-b3e6-95852c822b4e-kube-api-access-9v6mj\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428706 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428735 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.428760 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.429149 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735f137-8697-422b-b3e6-95852c822b4e-tmp\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.429546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.429421 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c735f137-8697-422b-b3e6-95852c822b4e-metrics-client-ca\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.431452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.431424 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.431719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.431695 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.431768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.431710 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.431993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.431480 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-grpc-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.432404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.432387 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.432483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.432393 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c735f137-8697-422b-b3e6-95852c822b4e-secret-thanos-querier-tls\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.452602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.452576 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v6mj\" (UniqueName: \"kubernetes.io/projected/c735f137-8697-422b-b3e6-95852c822b4e-kube-api-access-9v6mj\") pod \"thanos-querier-78dc685f45-bspxf\" (UID: \"c735f137-8697-422b-b3e6-95852c822b4e\") " pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.466867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.466839 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtwmg" event={"ID":"500e754b-8401-4604-8d32-5105317e9dc6","Type":"ContainerStarted","Data":"c146b5ef19806c7ff6d6b81fc770fa02b2bab762766793229b981b75b7513fd5"} Oct 30 17:12:59.466989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.466888 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gtwmg" event={"ID":"500e754b-8401-4604-8d32-5105317e9dc6","Type":"ContainerStarted","Data":"9399d93eb9bb7464c2d15f926f22a7fac8936b08652e7824b91af8fbefa68148"} Oct 30 17:12:59.468142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.468118 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"85a33ec817e5cbd4fedb978836ecf9ea43ea899fb15241401a99c93af4908eda"} Oct 30 17:12:59.507032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.506972 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gtwmg" podStartSLOduration=2.555248765 podStartE2EDuration="3.506959097s" totalImagesPullingTime="879.93492ms" totalInitContainerRuntime="71.775412ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:59.506785384 +0000 UTC m=+7.284257340" watchObservedRunningTime="2026-10-30 17:12:59.506959097 +0000 UTC m=+7.284431055" Oct 30 17:12:59.563032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.563002 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:12:59.873982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:12:59.873893 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78dc685f45-bspxf"] Oct 30 17:12:59.877873 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:12:59.877843 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc735f137_8697_422b_b3e6_95852c822b4e.slice/crio-a46e71f4e4b855d017ca1b0bf80bf00c9f65e73489a7e514eb6b21fd61cdf333 WatchSource:0}: Error finding container a46e71f4e4b855d017ca1b0bf80bf00c9f65e73489a7e514eb6b21fd61cdf333: Status 404 returned error can't find the container with id a46e71f4e4b855d017ca1b0bf80bf00c9f65e73489a7e514eb6b21fd61cdf333 Oct 30 17:13:00.474549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:00.474514 4622 generic.go:356] "Generic (PLEG): container finished" podID="ef900d92-5256-4606-8198-ee3bfb001552" containerID="9113be7414c8d9294e585afcd8c5796494ae61fee1d026fbd666f10de271f99a" exitCode=0 Oct 30 17:13:00.474926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:00.474554 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerDied","Data":"9113be7414c8d9294e585afcd8c5796494ae61fee1d026fbd666f10de271f99a"} Oct 30 17:13:00.476017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:00.475984 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"a46e71f4e4b855d017ca1b0bf80bf00c9f65e73489a7e514eb6b21fd61cdf333"} Oct 30 17:13:00.986213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:00.986180 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:13:00.987829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:00.986531 4622 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:13:01.454994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.454959 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f796dd7cf-fppn4"] Oct 30 17:13:01.457699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.457674 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.471398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471372 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 17:13:01.471398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471372 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 17:13:01.471592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471392 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:13:01.471592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471426 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 17:13:01.471592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471528 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4fg24uncuuedf\"" Oct 30 17:13:01.471592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.471570 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cwfvr\"" Oct 30 17:13:01.482853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.482781 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f796dd7cf-fppn4"] Oct 30 17:13:01.546192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546137 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2347c334-c94f-45a8-b1c3-e15d654d1862-audit-log\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546284 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-metrics-server-audit-profiles\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546332 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26qc\" (UniqueName: \"kubernetes.io/projected/2347c334-c94f-45a8-b1c3-e15d654d1862-kube-api-access-w26qc\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546372 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-client-certs\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546442 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-client-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546564 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.546663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.546624 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-tls\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647619 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2347c334-c94f-45a8-b1c3-e15d654d1862-audit-log\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647680 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-metrics-server-audit-profiles\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647704 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26qc\" (UniqueName: \"kubernetes.io/projected/2347c334-c94f-45a8-b1c3-e15d654d1862-kube-api-access-w26qc\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647732 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-client-certs\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647749 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-client-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647788 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.647840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.647840 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-tls\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.648087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.648069 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2347c334-c94f-45a8-b1c3-e15d654d1862-audit-log\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.649031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.648997 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-metrics-server-audit-profiles\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.649676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.649650 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2347c334-c94f-45a8-b1c3-e15d654d1862-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.650683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.650658 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-tls\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.651090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.651067 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-client-ca-bundle\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.651838 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.651814 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2347c334-c94f-45a8-b1c3-e15d654d1862-secret-metrics-server-client-certs\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.670976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.670950 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26qc\" (UniqueName: \"kubernetes.io/projected/2347c334-c94f-45a8-b1c3-e15d654d1862-kube-api-access-w26qc\") pod \"metrics-server-5f796dd7cf-fppn4\" (UID: \"2347c334-c94f-45a8-b1c3-e15d654d1862\") " pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:01.771942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:01.771858 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:02.082752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.082721 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f796dd7cf-fppn4"] Oct 30 17:13:02.087679 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:13:02.087630 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2347c334_c94f_45a8_b1c3_e15d654d1862.slice/crio-8570fd0a48d2703d7e9cad3d2a929af2d434ca77349fdc9b9d4b363b4a4fd2dc WatchSource:0}: Error finding container 8570fd0a48d2703d7e9cad3d2a929af2d434ca77349fdc9b9d4b363b4a4fd2dc: Status 404 returned error can't find the container with id 8570fd0a48d2703d7e9cad3d2a929af2d434ca77349fdc9b9d4b363b4a4fd2dc Oct 30 17:13:02.486118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.486073 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"66f40d765e1ff8fcbb27ac6783d437b2b083470b931c8df173eceebc1e3be8a4"} Oct 30 17:13:02.486636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.486151 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"62a11fbd0fe528d47db89964c56fd53ce9de32f93e251dceeff593133c62eba7"} Oct 30 17:13:02.486636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.486194 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"cb7f58527ae04ef2e01b9ac321405c30134f76de7e9a4f972ea1bb56621d97bf"} Oct 30 17:13:02.487427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.487398 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" event={"ID":"2347c334-c94f-45a8-b1c3-e15d654d1862","Type":"ContainerStarted","Data":"8570fd0a48d2703d7e9cad3d2a929af2d434ca77349fdc9b9d4b363b4a4fd2dc"} Oct 30 17:13:02.490908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.490880 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"b2dc1ad564e1606a5244e9e618a23b16b82977dc2e8a44ececa9eed96920aa9d"} Oct 30 17:13:02.491014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.490956 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"17238c2eb88f4f4eb7af5d156058b0a6dae65f0835aa3dd6d0b3f7e529e955c5"} Oct 30 17:13:02.491014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.490997 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"a7ca6db5bec01c29cdfc308c4cbda2f3cbfa175e6ca2369c74e2724d15c4e337"} Oct 30 17:13:02.491101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.491035 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"10447d61dc9f1d752e7a073211d030516160c0dde33d2a78fb4199e79e868741"} Oct 30 17:13:02.491101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:02.491072 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"c3901fb12668321e35607f1c627b40f5ab2e61eb47cde6a435a72a845110a5e2"} Oct 30 17:13:03.499476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.498997 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"9d2b68f7d7351a62f6a3850427f387aa62ee41520df9070015f9d6c7f5339775"} Oct 30 17:13:03.499476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.499114 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"6c85219e5318c68ac54b29a607ad7ce6610978b7f163d8e2fab1738cf40e9711"} Oct 30 17:13:03.499476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.499151 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" event={"ID":"c735f137-8697-422b-b3e6-95852c822b4e","Type":"ContainerStarted","Data":"660e4507afe5631fc17f291dd36d063ce9ba856b14c01db5e5cf7c5afc118c8a"} Oct 30 17:13:03.500242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.500221 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:13:03.503769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.503713 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ef900d92-5256-4606-8198-ee3bfb001552","Type":"ContainerStarted","Data":"8888a7b7f8c39ca0aa188290c124464051dd13774b04a4029b60ac0089acdc43"} Oct 30 17:13:03.589114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.589064 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" podStartSLOduration=1.899435615 podStartE2EDuration="4.589049201s" totalImagesPullingTime="2.689613586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:03.578290213 +0000 UTC m=+11.355762171" watchObservedRunningTime="2026-10-30 17:13:03.589049201 +0000 UTC m=+11.366521153" Oct 30 17:13:03.637745 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:03.637635 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.599469429 podStartE2EDuration="5.637616088s" totalImagesPullingTime="3.027005714s" totalInitContainerRuntime="11.140945ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:03.636050003 +0000 UTC m=+11.413521956" watchObservedRunningTime="2026-10-30 17:13:03.637616088 +0000 UTC m=+11.415088044" Oct 30 17:13:04.292109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.292078 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:13:04.295892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.295871 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:13:04.510166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.510125 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" event={"ID":"2347c334-c94f-45a8-b1c3-e15d654d1862","Type":"ContainerStarted","Data":"9e66392715d12025a8ec57f95d04152ea81951ce28f07ad63e27fe5ddbe896fc"} Oct 30 17:13:04.560280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.560023 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" podStartSLOduration=2.174221684 podStartE2EDuration="3.560009549s" totalImagesPullingTime="1.385787865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:04.559510367 +0000 UTC m=+12.336982318" watchObservedRunningTime="2026-10-30 17:13:04.560009549 +0000 UTC m=+12.337481504" Oct 30 17:13:04.741866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.741810 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:04.744184 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.744161 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.777515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.777483 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:04.879935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.879847 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.879935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.879906 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48lh\" (UniqueName: \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880010 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880072 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880142 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880235 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880310 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.880350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.880340 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.981780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981742 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.981780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981781 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981803 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981826 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48lh\" (UniqueName: \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981850 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981883 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981905 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.981953 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982443 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.982416 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.982604 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.982713 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.982823 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.982899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.982857 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.984916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.984884 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:04.985019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:04.984980 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:05.003344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.003316 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48lh\" (UniqueName: \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\") pod \"console-7dd969f75d-cqstz\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:05.054726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.054699 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:05.346270 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.346225 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:05.350818 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:13:05.350790 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b49e7e_e246_4194_94dd_9d9176325464.slice/crio-7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c WatchSource:0}: Error finding container 7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c: Status 404 returned error can't find the container with id 7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c Oct 30 17:13:05.515447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.515410 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd969f75d-cqstz" event={"ID":"f6b49e7e-e246-4194-94dd-9d9176325464","Type":"ContainerStarted","Data":"18c9c6f7a378fdba223c8d205d714c5cd37ba4690b2c664be362e567434a4fbc"} Oct 30 17:13:05.515837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.515468 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd969f75d-cqstz" event={"ID":"f6b49e7e-e246-4194-94dd-9d9176325464","Type":"ContainerStarted","Data":"7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c"} Oct 30 17:13:05.560774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.560722 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7dd969f75d-cqstz" podStartSLOduration=1.5607068050000001 podStartE2EDuration="1.560706805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:05.559885758 +0000 UTC m=+13.337357714" watchObservedRunningTime="2026-10-30 17:13:05.560706805 +0000 UTC m=+13.338178757" Oct 30 17:13:05.662391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:05.662315 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rzvbg" Oct 30 17:13:06.135586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:06.135553 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:10.346319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.346282 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:10.377339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.377307 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:13:10.380575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.380552 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.402713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.402668 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:13:10.521917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.521892 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78dc685f45-bspxf" Oct 30 17:13:10.527767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527747 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgbt\" (UniqueName: \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.527820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527787 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.527863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527817 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.527918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527898 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.527961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527926 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.527961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527941 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.528044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.527978 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.528044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.528004 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.628969 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629004 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629022 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629148 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629211 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629497 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgbt\" (UniqueName: \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629743 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629724 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629755 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629775 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.629927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.629859 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.630062 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.630041 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.630417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.630397 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.632069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.632051 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.632183 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.632167 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.651297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.651251 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgbt\" (UniqueName: \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\") pod \"console-7bd547bd9d-6fs99\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.691325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.691277 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:10.982579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:10.982497 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:13:10.986589 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:13:10.986562 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6110e6d_ecb4_4e29_8d9c_87440d92c80b.slice/crio-d3006b9834497a4cb72f781e5c337f4f8efa543e7c7bea0f8a7d0e43c9f42632 WatchSource:0}: Error finding container d3006b9834497a4cb72f781e5c337f4f8efa543e7c7bea0f8a7d0e43c9f42632: Status 404 returned error can't find the container with id d3006b9834497a4cb72f781e5c337f4f8efa543e7c7bea0f8a7d0e43c9f42632 Oct 30 17:13:11.540860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:11.540819 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd547bd9d-6fs99" event={"ID":"a6110e6d-ecb4-4e29-8d9c-87440d92c80b","Type":"ContainerStarted","Data":"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df"} Oct 30 17:13:11.541244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:11.540881 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd547bd9d-6fs99" event={"ID":"a6110e6d-ecb4-4e29-8d9c-87440d92c80b","Type":"ContainerStarted","Data":"d3006b9834497a4cb72f781e5c337f4f8efa543e7c7bea0f8a7d0e43c9f42632"} Oct 30 17:13:11.582047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:11.581995 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bd547bd9d-6fs99" podStartSLOduration=1.581982581 podStartE2EDuration="1.581982581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:11.581438963 +0000 UTC m=+19.358910917" watchObservedRunningTime="2026-10-30 17:13:11.581982581 +0000 UTC m=+19.359454536" Oct 30 17:13:12.846097 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:12.846065 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb9j4" Oct 30 17:13:15.055562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:15.055521 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:19.812110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:19.812075 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:20.692400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:20.692366 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:20.692587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:20.692423 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:20.697105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:20.697082 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:21.582826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:21.582800 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:13:21.666446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:21.666411 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:13:24.442401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:24.442372 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wlnpk" Oct 30 17:13:24.537076 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:13:24.537043 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:13:26.141750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:26.141723 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:26.145392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:26.145368 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f796dd7cf-fppn4" Oct 30 17:13:35.365712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.365659 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7dd969f75d-cqstz" podUID="f6b49e7e-e246-4194-94dd-9d9176325464" containerName="console" containerID="cri-o://18c9c6f7a378fdba223c8d205d714c5cd37ba4690b2c664be362e567434a4fbc" gracePeriod=15 Oct 30 17:13:35.629474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.629400 4622 generic.go:356] "Generic (PLEG): container finished" podID="f6b49e7e-e246-4194-94dd-9d9176325464" containerID="18c9c6f7a378fdba223c8d205d714c5cd37ba4690b2c664be362e567434a4fbc" exitCode=0 Oct 30 17:13:35.629609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.629474 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd969f75d-cqstz" event={"ID":"f6b49e7e-e246-4194-94dd-9d9176325464","Type":"ContainerDied","Data":"18c9c6f7a378fdba223c8d205d714c5cd37ba4690b2c664be362e567434a4fbc"} Oct 30 17:13:35.629609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.629522 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd969f75d-cqstz" event={"ID":"f6b49e7e-e246-4194-94dd-9d9176325464","Type":"ContainerDied","Data":"7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c"} Oct 30 17:13:35.629609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.629537 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f37484f5b289fed0ac1e9c05a537deeec408010d8ead0018f82eac3f49b695c" Oct 30 17:13:35.635622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.635599 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:35.732768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.732736 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.732960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.732779 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.732960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.732823 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.732960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.732861 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\" (UniqueName: \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.733156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733088 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.733156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733134 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.733283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733157 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.733283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733172 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.733283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733278 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\") pod \"f6b49e7e-e246-4194-94dd-9d9176325464\" (UID: \"f6b49e7e-e246-4194-94dd-9d9176325464\") " Oct 30 17:13:35.733452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733358 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.733507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733488 4622 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-trusted-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.733507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733502 4622 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-oauth-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.733613 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733501 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:35.733613 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733511 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.733714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.733617 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:35.735187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.735158 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "kube-api-access-q48lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:35.735317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.735194 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:35.735317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.735232 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config" pod "f6b49e7e-e246-4194-94dd-9d9176325464" (UID: "f6b49e7e-e246-4194-94dd-9d9176325464"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:35.817323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.817295 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-774c8b44ff-w4tz5"] Oct 30 17:13:35.817639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.817601 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f6b49e7e-e246-4194-94dd-9d9176325464" containerName="console" Oct 30 17:13:35.817700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.817641 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b49e7e-e246-4194-94dd-9d9176325464" containerName="console" Oct 30 17:13:35.817749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.817707 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b49e7e-e246-4194-94dd-9d9176325464" containerName="console" Oct 30 17:13:35.834679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834650 4622 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.834679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834678 4622 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b49e7e-e246-4194-94dd-9d9176325464-console-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.834879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834692 4622 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.834879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834709 4622 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b49e7e-e246-4194-94dd-9d9176325464-tmp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.834879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834724 4622 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b49e7e-e246-4194-94dd-9d9176325464-console-oauth-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.834879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.834738 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q48lh\" (UniqueName: \"kubernetes.io/projected/f6b49e7e-e246-4194-94dd-9d9176325464-kube-api-access-q48lh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:35.844602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.844571 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c8b44ff-w4tz5"] Oct 30 17:13:35.844774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.844694 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935484 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-service-ca\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935530 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-oauth-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935547 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt88f\" (UniqueName: \"kubernetes.io/projected/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-kube-api-access-lt88f\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935651 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-oauth-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935682 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-trusted-ca-bundle\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935744 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935813 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:35.935835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:35.935830 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-tmp\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.036887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036851 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-tmp\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.036887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036891 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-service-ca\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036923 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-oauth-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036937 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt88f\" (UniqueName: \"kubernetes.io/projected/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-kube-api-access-lt88f\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036967 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-oauth-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.036990 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-trusted-ca-bundle\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037047 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037095 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037375 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-tmp\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037823 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-service-ca\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037820 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-oauth-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.037990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.037947 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.038106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.038079 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-trusted-ca-bundle\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.039632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.039603 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-oauth-config\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.039728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.039685 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-console-serving-cert\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.058216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.058192 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt88f\" (UniqueName: \"kubernetes.io/projected/d24edaf4-dbf7-4fbb-96e6-0f6275e04572-kube-api-access-lt88f\") pod \"console-774c8b44ff-w4tz5\" (UID: \"d24edaf4-dbf7-4fbb-96e6-0f6275e04572\") " pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.166417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.166384 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:36.457783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.457753 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c8b44ff-w4tz5"] Oct 30 17:13:36.462613 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:13:36.462554 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24edaf4_dbf7_4fbb_96e6_0f6275e04572.slice/crio-3fa5c7f1f5994a1f02db389442255edd10fc9ad0083be5ca23ce1386ff2cc938 WatchSource:0}: Error finding container 3fa5c7f1f5994a1f02db389442255edd10fc9ad0083be5ca23ce1386ff2cc938: Status 404 returned error can't find the container with id 3fa5c7f1f5994a1f02db389442255edd10fc9ad0083be5ca23ce1386ff2cc938 Oct 30 17:13:36.634982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.634948 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c8b44ff-w4tz5" event={"ID":"d24edaf4-dbf7-4fbb-96e6-0f6275e04572","Type":"ContainerStarted","Data":"1e8158af567d7aae20da4e69126f1651cae2bd6956dd36e36ff5b902b8ab4127"} Oct 30 17:13:36.634982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.634978 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd969f75d-cqstz" Oct 30 17:13:36.635179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.635000 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c8b44ff-w4tz5" event={"ID":"d24edaf4-dbf7-4fbb-96e6-0f6275e04572","Type":"ContainerStarted","Data":"3fa5c7f1f5994a1f02db389442255edd10fc9ad0083be5ca23ce1386ff2cc938"} Oct 30 17:13:36.677512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.677461 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-774c8b44ff-w4tz5" podStartSLOduration=1.67744693 podStartE2EDuration="1.67744693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:36.677412102 +0000 UTC m=+44.454884057" watchObservedRunningTime="2026-10-30 17:13:36.67744693 +0000 UTC m=+44.454918879" Oct 30 17:13:36.718206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.718122 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:36.723159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:36.723124 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dd969f75d-cqstz"] Oct 30 17:13:38.371209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:38.371174 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b49e7e-e246-4194-94dd-9d9176325464" path="/var/lib/kubelet/pods/f6b49e7e-e246-4194-94dd-9d9176325464/volumes" Oct 30 17:13:46.167400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.167359 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:46.167824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.167443 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:46.172445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.172422 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:46.675765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.675733 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774c8b44ff-w4tz5" Oct 30 17:13:46.686187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.686141 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58cd6766d5-btnxw" podUID="6723eff8-01c0-484c-88eb-7f363b0e565a" containerName="console" containerID="cri-o://34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4" gracePeriod=15 Oct 30 17:13:46.748544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.748512 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:13:46.926379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:46.926319 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:13:47.027362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027326 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027384 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027421 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027443 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027466 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027506 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027548 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027615 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") pod \"6723eff8-01c0-484c-88eb-7f363b0e565a\" (UID: \"6723eff8-01c0-484c-88eb-7f363b0e565a\") " Oct 30 17:13:47.027978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027956 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.028037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027968 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.028156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.027988 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.028156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.028130 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:47.028334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.028159 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.029802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.029780 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:47.029802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.029788 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "kube-api-access-v8s9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.029922 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.029873 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert" pod "6723eff8-01c0-484c-88eb-7f363b0e565a" (UID: "6723eff8-01c0-484c-88eb-7f363b0e565a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:47.128229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128193 4622 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-oauth-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128225 4622 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-console-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128236 4622 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128245 4622 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-trusted-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128255 4622 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6723eff8-01c0-484c-88eb-7f363b0e565a-tmp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128287 4622 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6723eff8-01c0-484c-88eb-7f363b0e565a-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128296 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8s9t\" (UniqueName: \"kubernetes.io/projected/6723eff8-01c0-484c-88eb-7f363b0e565a-kube-api-access-v8s9t\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.128477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.128304 4622 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6723eff8-01c0-484c-88eb-7f363b0e565a-console-oauth-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.675611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.675573 4622 generic.go:356] "Generic (PLEG): container finished" podID="6723eff8-01c0-484c-88eb-7f363b0e565a" containerID="34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4" exitCode=0 Oct 30 17:13:47.676055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.675618 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cd6766d5-btnxw" Oct 30 17:13:47.676055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.675660 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd6766d5-btnxw" event={"ID":"6723eff8-01c0-484c-88eb-7f363b0e565a","Type":"ContainerDied","Data":"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4"} Oct 30 17:13:47.676055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.675707 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cd6766d5-btnxw" event={"ID":"6723eff8-01c0-484c-88eb-7f363b0e565a","Type":"ContainerDied","Data":"c8dd9b77ea96aae52736edfdfcea090f9220f790f0bbb493ea86717f2f6cd0e4"} Oct 30 17:13:47.676055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.675742 4622 scope.go:122] "RemoveContainer" containerID="34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4" Oct 30 17:13:47.684424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.684406 4622 scope.go:122] "RemoveContainer" containerID="34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4" Oct 30 17:13:47.684712 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:13:47.684692 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4\": container with ID starting with 34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4 not found: ID does not exist" containerID="34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4" Oct 30 17:13:47.684782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.684721 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4"} err="failed to get container status \"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4\": rpc error: code = NotFound desc = could not find container \"34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4\": container with ID starting with 34daf0057d3e9596c5f1b2ba4160529898a3af764d2388bedcea9efa81a4d2f4 not found: ID does not exist" Oct 30 17:13:47.720347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.720314 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:13:47.730333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:47.730293 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58cd6766d5-btnxw"] Oct 30 17:13:48.370864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:13:48.370831 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6723eff8-01c0-484c-88eb-7f363b0e565a" path="/var/lib/kubelet/pods/6723eff8-01c0-484c-88eb-7f363b0e565a/volumes" Oct 30 17:13:54.533019 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:13:54.532980 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:14:11.768743 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:11.768684 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7bd547bd9d-6fs99" podUID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" containerName="console" containerID="cri-o://be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df" gracePeriod=15 Oct 30 17:14:11.998325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:11.998302 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:14:12.139616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139577 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139630 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\" (UniqueName: \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139659 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139679 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139716 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139743 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.139795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139775 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.140089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.139813 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\") pod \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\" (UID: \"a6110e6d-ecb4-4e29-8d9c-87440d92c80b\") " Oct 30 17:14:12.140155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.140101 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:12.140382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.140328 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:12.140382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.140329 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:12.140382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.140365 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:12.140617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.140429 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:12.141814 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.141794 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:12.141971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.141949 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:12.142014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.141956 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt" pod "a6110e6d-ecb4-4e29-8d9c-87440d92c80b" (UID: "a6110e6d-ecb4-4e29-8d9c-87440d92c80b"). InnerVolumeSpecName "kube-api-access-8fgbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:12.241366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241326 4622 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-tmp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241357 4622 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-trusted-ca-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241368 4622 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-oauth-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241380 4622 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-oauth-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241389 4622 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-serving-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241397 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgbt\" (UniqueName: \"kubernetes.io/projected/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-kube-api-access-8fgbt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241406 4622 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.241639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.241413 4622 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6110e6d-ecb4-4e29-8d9c-87440d92c80b-console-config\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:14:12.762321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.762285 4622 generic.go:356] "Generic (PLEG): container finished" podID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" containerID="be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df" exitCode=0 Oct 30 17:14:12.762510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.762328 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd547bd9d-6fs99" Oct 30 17:14:12.762510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.762365 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd547bd9d-6fs99" event={"ID":"a6110e6d-ecb4-4e29-8d9c-87440d92c80b","Type":"ContainerDied","Data":"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df"} Oct 30 17:14:12.762510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.762416 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd547bd9d-6fs99" event={"ID":"a6110e6d-ecb4-4e29-8d9c-87440d92c80b","Type":"ContainerDied","Data":"d3006b9834497a4cb72f781e5c337f4f8efa543e7c7bea0f8a7d0e43c9f42632"} Oct 30 17:14:12.762510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.762435 4622 scope.go:122] "RemoveContainer" containerID="be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df" Oct 30 17:14:12.770221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.770026 4622 scope.go:122] "RemoveContainer" containerID="be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df" Oct 30 17:14:12.770481 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:14:12.770303 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df\": container with ID starting with be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df not found: ID does not exist" containerID="be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df" Oct 30 17:14:12.770481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.770329 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df"} err="failed to get container status \"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df\": rpc error: code = NotFound desc = could not find container \"be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df\": container with ID starting with be6c3f2f8ce729a15cf0eb0db8c0fb4dcbbd7674c13d181a8fa830c40599c0df not found: ID does not exist" Oct 30 17:14:12.806525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.806492 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:14:12.816666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:12.816641 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd547bd9d-6fs99"] Oct 30 17:14:14.371193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:14:14.371154 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" path="/var/lib/kubelet/pods/a6110e6d-ecb4-4e29-8d9c-87440d92c80b/volumes" Oct 30 17:14:24.532583 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:14:24.532541 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:14:54.532876 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:14:54.532836 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:15:24.532538 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:15:24.532503 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:15:54.532503 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:15:54.532462 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:16:04.202119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202088 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd"] Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202407 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" containerName="console" Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202422 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" containerName="console" Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202446 4622 cpu_manager.go:414] "containerMap: removing container" podUID="6723eff8-01c0-484c-88eb-7f363b0e565a" containerName="console" Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202454 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6723eff8-01c0-484c-88eb-7f363b0e565a" containerName="console" Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202530 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="6723eff8-01c0-484c-88eb-7f363b0e565a" containerName="console" Oct 30 17:16:04.202675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.202547 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6110e6d-ecb4-4e29-8d9c-87440d92c80b" containerName="console" Oct 30 17:16:04.204454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.204436 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.227698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.227675 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:16:04.227843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.227713 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-r7lj5\"" Oct 30 17:16:04.227843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.227744 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:16:04.244919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.244890 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd"] Oct 30 17:16:04.265763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.265728 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.265940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.265777 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.265940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.265796 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7czt\" (UniqueName: \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.366346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.366307 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.366511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.366369 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.366511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.366400 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7czt\" (UniqueName: \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.366741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.366718 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.366741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.366730 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.388564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.388535 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7czt\" (UniqueName: \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.516294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.516155 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:04.808916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:04.808877 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd"] Oct 30 17:16:04.813243 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:16:04.813218 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbead01be_8436_475c_b412_1504a7f3aa96.slice/crio-ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e WatchSource:0}: Error finding container ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e: Status 404 returned error can't find the container with id ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e Oct 30 17:16:05.153316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:05.153216 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" event={"ID":"bead01be-8436-475c-b412-1504a7f3aa96","Type":"ContainerStarted","Data":"ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e"} Oct 30 17:16:10.175398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:10.175358 4622 generic.go:356] "Generic (PLEG): container finished" podID="bead01be-8436-475c-b412-1504a7f3aa96" containerID="1df5a34823b23dee8e46fbc9a785e53e2ebba810b8f39eefbed18f6673cef9e3" exitCode=0 Oct 30 17:16:10.175787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:10.175408 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" event={"ID":"bead01be-8436-475c-b412-1504a7f3aa96","Type":"ContainerDied","Data":"1df5a34823b23dee8e46fbc9a785e53e2ebba810b8f39eefbed18f6673cef9e3"} Oct 30 17:16:23.221792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:23.221759 4622 generic.go:356] "Generic (PLEG): container finished" podID="bead01be-8436-475c-b412-1504a7f3aa96" containerID="7bcd87a3a6de11093425c0907b6897ce742dd0d134d8b3cb063c0bfa2da7b5c2" exitCode=0 Oct 30 17:16:23.222318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:23.221825 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" event={"ID":"bead01be-8436-475c-b412-1504a7f3aa96","Type":"ContainerDied","Data":"7bcd87a3a6de11093425c0907b6897ce742dd0d134d8b3cb063c0bfa2da7b5c2"} Oct 30 17:16:24.538867 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:16:24.538832 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:16:29.246107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:29.246067 4622 generic.go:356] "Generic (PLEG): container finished" podID="bead01be-8436-475c-b412-1504a7f3aa96" containerID="be408382feb4d55667ec08902b5951a221621029b1acfea19c3f9cfc86b700e7" exitCode=0 Oct 30 17:16:29.246526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:29.246128 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" event={"ID":"bead01be-8436-475c-b412-1504a7f3aa96","Type":"ContainerDied","Data":"be408382feb4d55667ec08902b5951a221621029b1acfea19c3f9cfc86b700e7"} Oct 30 17:16:30.381339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.381307 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:30.482177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.482136 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\" (UniqueName: \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\") pod \"bead01be-8436-475c-b412-1504a7f3aa96\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " Oct 30 17:16:30.482406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.482204 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\") pod \"bead01be-8436-475c-b412-1504a7f3aa96\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " Oct 30 17:16:30.482406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.482230 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\") pod \"bead01be-8436-475c-b412-1504a7f3aa96\" (UID: \"bead01be-8436-475c-b412-1504a7f3aa96\") " Oct 30 17:16:30.482814 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.482788 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle" pod "bead01be-8436-475c-b412-1504a7f3aa96" (UID: "bead01be-8436-475c-b412-1504a7f3aa96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.484432 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.484409 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt" pod "bead01be-8436-475c-b412-1504a7f3aa96" (UID: "bead01be-8436-475c-b412-1504a7f3aa96"). InnerVolumeSpecName "kube-api-access-s7czt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:30.487824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.487795 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util" pod "bead01be-8436-475c-b412-1504a7f3aa96" (UID: "bead01be-8436-475c-b412-1504a7f3aa96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:30.583081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.583050 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.583081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.583079 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7czt\" (UniqueName: \"kubernetes.io/projected/bead01be-8436-475c-b412-1504a7f3aa96-kube-api-access-s7czt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:30.583288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:30.583089 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bead01be-8436-475c-b412-1504a7f3aa96-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:31.254709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:31.254685 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" Oct 30 17:16:31.254878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:31.254683 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7rlbd" event={"ID":"bead01be-8436-475c-b412-1504a7f3aa96","Type":"ContainerDied","Data":"ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e"} Oct 30 17:16:31.254878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:31.254798 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8cceecaed6506fabe8d4d1b9151c058db96337fa2ac8048a5c3bbb9a6a038e" Oct 30 17:16:37.508709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.508670 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw"] Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.508986 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="extract" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.508999 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="extract" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.509023 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="pull" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.509030 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="pull" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.509051 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="util" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.509056 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="util" Oct 30 17:16:37.509253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.509119 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bead01be-8436-475c-b412-1504a7f3aa96" containerName="extract" Oct 30 17:16:37.510993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.510974 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.536368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.536339 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 17:16:37.536368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.536340 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:16:37.536547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.536406 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bkw5q\"" Oct 30 17:16:37.537488 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.537468 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw"] Oct 30 17:16:37.643593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.643561 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02730837-4dbd-4b4f-b96b-3061fab5738a-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.643764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.643605 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txp86\" (UniqueName: \"kubernetes.io/projected/02730837-4dbd-4b4f-b96b-3061fab5738a-kube-api-access-txp86\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.745015 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.744980 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02730837-4dbd-4b4f-b96b-3061fab5738a-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.745199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.745026 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txp86\" (UniqueName: \"kubernetes.io/projected/02730837-4dbd-4b4f-b96b-3061fab5738a-kube-api-access-txp86\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.745396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.745375 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02730837-4dbd-4b4f-b96b-3061fab5738a-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.766331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.766256 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txp86\" (UniqueName: \"kubernetes.io/projected/02730837-4dbd-4b4f-b96b-3061fab5738a-kube-api-access-txp86\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-cg9sw\" (UID: \"02730837-4dbd-4b4f-b96b-3061fab5738a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:37.825402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:37.825366 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" Oct 30 17:16:38.151227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:38.151193 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw"] Oct 30 17:16:38.155842 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:16:38.155810 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02730837_4dbd_4b4f_b96b_3061fab5738a.slice/crio-ce1616dd185cce7630fc3cc651ce84ba87c83f458dc0c6e6c4edc3f454282d05 WatchSource:0}: Error finding container ce1616dd185cce7630fc3cc651ce84ba87c83f458dc0c6e6c4edc3f454282d05: Status 404 returned error can't find the container with id ce1616dd185cce7630fc3cc651ce84ba87c83f458dc0c6e6c4edc3f454282d05 Oct 30 17:16:38.281424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:38.281391 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" event={"ID":"02730837-4dbd-4b4f-b96b-3061fab5738a","Type":"ContainerStarted","Data":"ce1616dd185cce7630fc3cc651ce84ba87c83f458dc0c6e6c4edc3f454282d05"} Oct 30 17:16:40.292042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:40.292009 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" event={"ID":"02730837-4dbd-4b4f-b96b-3061fab5738a","Type":"ContainerStarted","Data":"b7395d7e0d231636c221fc9a2b384ffe11706ca0bb08912fea84c3b1bfc9eb5f"} Oct 30 17:16:40.356910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:40.356840 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-cg9sw" podStartSLOduration=1.944700601 podStartE2EDuration="3.356820215s" totalImagesPullingTime="1.412119614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:40.345305895 +0000 UTC m=+228.122777852" watchObservedRunningTime="2026-10-30 17:16:40.356820215 +0000 UTC m=+228.134292173" Oct 30 17:16:42.008148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.008113 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v"] Oct 30 17:16:42.010678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.010656 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.033420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.033383 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:16:42.033420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.033383 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:16:42.033784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.033755 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-7pmj4\"" Oct 30 17:16:42.035271 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.035234 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v"] Oct 30 17:16:42.184400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.184366 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhwp\" (UniqueName: \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.184601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.184407 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.184601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.184513 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.285312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.285281 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.285453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.285323 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhwp\" (UniqueName: \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.285453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.285347 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.285719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.285703 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.285755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.285733 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.309560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.309533 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhwp\" (UniqueName: \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.323377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.323350 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:42.617772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:42.617674 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v"] Oct 30 17:16:42.621961 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:16:42.621929 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8952e6_c554_4890_868f_25ebbd2b79bc.slice/crio-53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc WatchSource:0}: Error finding container 53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc: Status 404 returned error can't find the container with id 53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc Oct 30 17:16:43.305348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.305317 4622 generic.go:356] "Generic (PLEG): container finished" podID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerID="5b5ced691eace89795e54fd7ffc41941313a63760a9ac475977f0ed8a1efe7e2" exitCode=0 Oct 30 17:16:43.305798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.305366 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" event={"ID":"af8952e6-c554-4890-868f-25ebbd2b79bc","Type":"ContainerDied","Data":"5b5ced691eace89795e54fd7ffc41941313a63760a9ac475977f0ed8a1efe7e2"} Oct 30 17:16:43.305798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.305401 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" event={"ID":"af8952e6-c554-4890-868f-25ebbd2b79bc","Type":"ContainerStarted","Data":"53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc"} Oct 30 17:16:43.564631 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.564527 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88"] Oct 30 17:16:43.566906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.566884 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.580634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.580613 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zmrgl\"" Oct 30 17:16:43.580806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.580755 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 17:16:43.580887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.580811 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 17:16:43.589494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.589467 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88"] Oct 30 17:16:43.595713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.595688 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zszk\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-kube-api-access-6zszk\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.595813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.595738 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.696367 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.696334 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zszk\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-kube-api-access-6zszk\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.696539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.696389 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.717587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.717554 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.717773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.717750 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zszk\" (UniqueName: \"kubernetes.io/projected/a88117ca-c3a4-4178-869b-f291c80bd2d8-kube-api-access-6zszk\") pod \"cert-manager-cainjector-5c57bbd4fc-4sb88\" (UID: \"a88117ca-c3a4-4178-869b-f291c80bd2d8\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:43.887458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:43.887379 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" Oct 30 17:16:44.177920 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:44.177822 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88"] Oct 30 17:16:44.182142 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:16:44.182104 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda88117ca_c3a4_4178_869b_f291c80bd2d8.slice/crio-e9145f6ab56d558ed3710b1d0d01d68846f55425fc3443c0a7c1de384715ae04 WatchSource:0}: Error finding container e9145f6ab56d558ed3710b1d0d01d68846f55425fc3443c0a7c1de384715ae04: Status 404 returned error can't find the container with id e9145f6ab56d558ed3710b1d0d01d68846f55425fc3443c0a7c1de384715ae04 Oct 30 17:16:44.312382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:44.312345 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" event={"ID":"a88117ca-c3a4-4178-869b-f291c80bd2d8","Type":"ContainerStarted","Data":"e9145f6ab56d558ed3710b1d0d01d68846f55425fc3443c0a7c1de384715ae04"} Oct 30 17:16:46.323670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:46.323637 4622 generic.go:356] "Generic (PLEG): container finished" podID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerID="0475068fdf7ca39a9677d4a03f42c935ac5a7916415ee8667430fa31fdfdfe03" exitCode=0 Oct 30 17:16:46.324134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:46.323716 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" event={"ID":"af8952e6-c554-4890-868f-25ebbd2b79bc","Type":"ContainerDied","Data":"0475068fdf7ca39a9677d4a03f42c935ac5a7916415ee8667430fa31fdfdfe03"} Oct 30 17:16:47.330216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.330179 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" event={"ID":"a88117ca-c3a4-4178-869b-f291c80bd2d8","Type":"ContainerStarted","Data":"0e1d2804062f95475993f2930484bfa54a85392d94b3937bbbcafbef834649ef"} Oct 30 17:16:47.332689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.332661 4622 generic.go:356] "Generic (PLEG): container finished" podID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerID="24b2ba401ec63f9f4c6f45a8b122946734cf56ca9933b1244629100434b91e22" exitCode=0 Oct 30 17:16:47.332826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.332713 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" event={"ID":"af8952e6-c554-4890-868f-25ebbd2b79bc","Type":"ContainerDied","Data":"24b2ba401ec63f9f4c6f45a8b122946734cf56ca9933b1244629100434b91e22"} Oct 30 17:16:47.366783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.366733 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4sb88" podStartSLOduration=1.6062686 podStartE2EDuration="4.366720075s" totalImagesPullingTime="2.760451475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:47.366036293 +0000 UTC m=+235.143508260" watchObservedRunningTime="2026-10-30 17:16:47.366720075 +0000 UTC m=+235.144192021" Oct 30 17:16:47.584527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.584442 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5"] Oct 30 17:16:47.587067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.587048 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.610386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.610359 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5"] Oct 30 17:16:47.631799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.631769 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.631936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.631809 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstjg\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-kube-api-access-rstjg\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.697572 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.697545 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t9v4q\"" Oct 30 17:16:47.733160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.733122 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.733160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.733161 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstjg\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-kube-api-access-rstjg\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.754605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.754573 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.754754 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.754617 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstjg\" (UniqueName: \"kubernetes.io/projected/f11bdc67-cf8d-4b50-82da-4b0b6f380f92-kube-api-access-rstjg\") pod \"cert-manager-webhook-5dbbbf75cc-n5cq5\" (UID: \"f11bdc67-cf8d-4b50-82da-4b0b6f380f92\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:47.897103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:47.897010 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:48.187502 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.187412 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5"] Oct 30 17:16:48.191522 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:16:48.191493 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf11bdc67_cf8d_4b50_82da_4b0b6f380f92.slice/crio-c323bfe708fdd76162d406114ab1a73e916f17ad9951150f349a9cc08b0bd5ef WatchSource:0}: Error finding container c323bfe708fdd76162d406114ab1a73e916f17ad9951150f349a9cc08b0bd5ef: Status 404 returned error can't find the container with id c323bfe708fdd76162d406114ab1a73e916f17ad9951150f349a9cc08b0bd5ef Oct 30 17:16:48.338670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.338632 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" event={"ID":"f11bdc67-cf8d-4b50-82da-4b0b6f380f92","Type":"ContainerStarted","Data":"0829916ba106e215af7bdba503475b4f398f7e52516fe268621ae871724bbb1f"} Oct 30 17:16:48.339129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.338683 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" event={"ID":"f11bdc67-cf8d-4b50-82da-4b0b6f380f92","Type":"ContainerStarted","Data":"c323bfe708fdd76162d406114ab1a73e916f17ad9951150f349a9cc08b0bd5ef"} Oct 30 17:16:48.339129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.338828 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:48.375342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.375288 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" podStartSLOduration=1.375258473 podStartE2EDuration="1.375258473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:48.375204741 +0000 UTC m=+236.152676696" watchObservedRunningTime="2026-10-30 17:16:48.375258473 +0000 UTC m=+236.152730428" Oct 30 17:16:48.449006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.448981 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:48.539285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.539231 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\" (UniqueName: \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\") pod \"af8952e6-c554-4890-868f-25ebbd2b79bc\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " Oct 30 17:16:48.539438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.539317 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\") pod \"af8952e6-c554-4890-868f-25ebbd2b79bc\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " Oct 30 17:16:48.539438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.539352 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\") pod \"af8952e6-c554-4890-868f-25ebbd2b79bc\" (UID: \"af8952e6-c554-4890-868f-25ebbd2b79bc\") " Oct 30 17:16:48.539725 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.539694 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle" pod "af8952e6-c554-4890-868f-25ebbd2b79bc" (UID: "af8952e6-c554-4890-868f-25ebbd2b79bc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:48.541314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.541288 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp" pod "af8952e6-c554-4890-868f-25ebbd2b79bc" (UID: "af8952e6-c554-4890-868f-25ebbd2b79bc"). InnerVolumeSpecName "kube-api-access-kvhwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:48.543784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.543762 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util" pod "af8952e6-c554-4890-868f-25ebbd2b79bc" (UID: "af8952e6-c554-4890-868f-25ebbd2b79bc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:48.640418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.640387 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:48.640418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.640413 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhwp\" (UniqueName: \"kubernetes.io/projected/af8952e6-c554-4890-868f-25ebbd2b79bc-kube-api-access-kvhwp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:48.640418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:48.640427 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af8952e6-c554-4890-868f-25ebbd2b79bc-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:16:49.344766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:49.344738 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" Oct 30 17:16:49.345173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:49.344764 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85f8l9v" event={"ID":"af8952e6-c554-4890-868f-25ebbd2b79bc","Type":"ContainerDied","Data":"53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc"} Oct 30 17:16:49.345173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:49.344808 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53869fbdde91927c79d39b1ff1f81fcd98e84347d3345e54d55f3f30117594cc" Oct 30 17:16:54.347576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:54.347541 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n5cq5" Oct 30 17:16:54.532914 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:16:54.532878 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:16:59.799402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799371 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx"] Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799660 4622 cpu_manager.go:414] "containerMap: removing container" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="pull" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799671 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="pull" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799685 4622 cpu_manager.go:414] "containerMap: removing container" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="extract" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799690 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="extract" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799699 4622 cpu_manager.go:414] "containerMap: removing container" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="util" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799705 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="util" Oct 30 17:16:59.812113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.799810 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="af8952e6-c554-4890-868f-25ebbd2b79bc" containerName="extract" Oct 30 17:16:59.818041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.818020 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:16:59.823655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.823627 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx"] Oct 30 17:16:59.832590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.832565 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:16:59.832590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.832565 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 17:16:59.832748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.832611 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gdzz9\"" Oct 30 17:16:59.931635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.931600 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ebcc1da-f8b6-40be-913c-ca658edd3bea-tmp\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:16:59.931635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:16:59.931641 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zll4f\" (UniqueName: \"kubernetes.io/projected/7ebcc1da-f8b6-40be-913c-ca658edd3bea-kube-api-access-zll4f\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.032678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.032646 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ebcc1da-f8b6-40be-913c-ca658edd3bea-tmp\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.032856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.032688 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zll4f\" (UniqueName: \"kubernetes.io/projected/7ebcc1da-f8b6-40be-913c-ca658edd3bea-kube-api-access-zll4f\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.033022 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.033002 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ebcc1da-f8b6-40be-913c-ca658edd3bea-tmp\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.054157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.054077 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zll4f\" (UniqueName: \"kubernetes.io/projected/7ebcc1da-f8b6-40be-913c-ca658edd3bea-kube-api-access-zll4f\") pod \"openshift-lws-operator-8465994787-5fmgx\" (UID: \"7ebcc1da-f8b6-40be-913c-ca658edd3bea\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.144188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.144155 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" Oct 30 17:17:00.453174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:00.453091 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx"] Oct 30 17:17:00.457159 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:00.457130 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ebcc1da_f8b6_40be_913c_ca658edd3bea.slice/crio-746c828eedfca4312dba20594e05cbceb67af246500f500da1f8ea6ad375f09b WatchSource:0}: Error finding container 746c828eedfca4312dba20594e05cbceb67af246500f500da1f8ea6ad375f09b: Status 404 returned error can't find the container with id 746c828eedfca4312dba20594e05cbceb67af246500f500da1f8ea6ad375f09b Oct 30 17:17:01.391204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:01.391165 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" event={"ID":"7ebcc1da-f8b6-40be-913c-ca658edd3bea","Type":"ContainerStarted","Data":"746c828eedfca4312dba20594e05cbceb67af246500f500da1f8ea6ad375f09b"} Oct 30 17:17:02.397473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:02.397438 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" event={"ID":"7ebcc1da-f8b6-40be-913c-ca658edd3bea","Type":"ContainerStarted","Data":"065ab391ef4b8fb93e60e1090d503477e9c726b8c64129e3b4a95e1208d0b8fa"} Oct 30 17:17:02.435753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:02.435668 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5fmgx" podStartSLOduration=1.876857542 podStartE2EDuration="3.435652212s" totalImagesPullingTime="1.55879467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:02.434550876 +0000 UTC m=+250.212022830" watchObservedRunningTime="2026-10-30 17:17:02.435652212 +0000 UTC m=+250.213124168" Oct 30 17:17:07.596089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.596012 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4"] Oct 30 17:17:07.598835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.598814 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.611557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.611539 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:17:07.621890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.621868 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:17:07.622063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.622037 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-hbxsm\"" Oct 30 17:17:07.623497 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.623478 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4"] Oct 30 17:17:07.699037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.699005 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.699037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.699053 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.699247 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.699089 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.800465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.800429 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.800607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.800488 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.800607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.800523 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.800895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.800876 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.800936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.800881 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.821953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.821921 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:07.911015 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:07.910930 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:08.200019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:08.199893 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4"] Oct 30 17:17:08.205104 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:08.205075 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516276d9_db24_423c_a834_4c2cfe515890.slice/crio-9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44 WatchSource:0}: Error finding container 9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44: Status 404 returned error can't find the container with id 9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44 Oct 30 17:17:08.422354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:08.422324 4622 generic.go:356] "Generic (PLEG): container finished" podID="516276d9-db24-423c-a834-4c2cfe515890" containerID="9d3f1caafaa03543b8740831031d298895b259d00b5e0b47b2ca678f447345c9" exitCode=0 Oct 30 17:17:08.422525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:08.422385 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" event={"ID":"516276d9-db24-423c-a834-4c2cfe515890","Type":"ContainerDied","Data":"9d3f1caafaa03543b8740831031d298895b259d00b5e0b47b2ca678f447345c9"} Oct 30 17:17:08.422525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:08.422439 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" event={"ID":"516276d9-db24-423c-a834-4c2cfe515890","Type":"ContainerStarted","Data":"9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44"} Oct 30 17:17:09.428881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:09.428845 4622 generic.go:356] "Generic (PLEG): container finished" podID="516276d9-db24-423c-a834-4c2cfe515890" containerID="9dd1e7265de4a70784e67e965de4407344809b8abdec41f0c37a42309cbe2942" exitCode=0 Oct 30 17:17:09.429319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:09.428897 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" event={"ID":"516276d9-db24-423c-a834-4c2cfe515890","Type":"ContainerDied","Data":"9dd1e7265de4a70784e67e965de4407344809b8abdec41f0c37a42309cbe2942"} Oct 30 17:17:10.437323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:10.436562 4622 generic.go:356] "Generic (PLEG): container finished" podID="516276d9-db24-423c-a834-4c2cfe515890" containerID="b908209368539cf59758aaa447e9c77f289028c66392ee62324c99679805463c" exitCode=0 Oct 30 17:17:10.437323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:10.436608 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" event={"ID":"516276d9-db24-423c-a834-4c2cfe515890","Type":"ContainerDied","Data":"b908209368539cf59758aaa447e9c77f289028c66392ee62324c99679805463c"} Oct 30 17:17:11.561575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.561547 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:11.635674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.635638 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\") pod \"516276d9-db24-423c-a834-4c2cfe515890\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " Oct 30 17:17:11.635845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.635717 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\") pod \"516276d9-db24-423c-a834-4c2cfe515890\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " Oct 30 17:17:11.635845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.635796 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\") pod \"516276d9-db24-423c-a834-4c2cfe515890\" (UID: \"516276d9-db24-423c-a834-4c2cfe515890\") " Oct 30 17:17:11.636331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.636296 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle" pod "516276d9-db24-423c-a834-4c2cfe515890" (UID: "516276d9-db24-423c-a834-4c2cfe515890"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.637723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.637702 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk" pod "516276d9-db24-423c-a834-4c2cfe515890" (UID: "516276d9-db24-423c-a834-4c2cfe515890"). InnerVolumeSpecName "kube-api-access-pmjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:11.640917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.640886 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util" pod "516276d9-db24-423c-a834-4c2cfe515890" (UID: "516276d9-db24-423c-a834-4c2cfe515890"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.737296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.737182 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.737296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.737212 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/516276d9-db24-423c-a834-4c2cfe515890-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.737296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:11.737225 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjzk\" (UniqueName: \"kubernetes.io/projected/516276d9-db24-423c-a834-4c2cfe515890-kube-api-access-pmjzk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:17:12.447391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:12.447352 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" event={"ID":"516276d9-db24-423c-a834-4c2cfe515890","Type":"ContainerDied","Data":"9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44"} Oct 30 17:17:12.447391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:12.447375 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71clnr4" Oct 30 17:17:12.447604 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:12.447406 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9927f96924be6ba625771d4b497ffbef51b2b55eec55e37301de28c17ae01f44" Oct 30 17:17:17.621977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.621942 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5"] Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622237 4622 cpu_manager.go:414] "containerMap: removing container" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="pull" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622247 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="pull" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622280 4622 cpu_manager.go:414] "containerMap: removing container" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="extract" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622285 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="extract" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622296 4622 cpu_manager.go:414] "containerMap: removing container" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="util" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622300 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="util" Oct 30 17:17:17.622419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.622362 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="516276d9-db24-423c-a834-4c2cfe515890" containerName="extract" Oct 30 17:17:17.624649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.624631 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.637698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.637672 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 17:17:17.648227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.648201 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 17:17:17.648349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.648244 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 17:17:17.648349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.648336 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 17:17:17.652820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.652806 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qkrcd\"" Oct 30 17:17:17.672618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.672581 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5"] Oct 30 17:17:17.737721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.737694 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 17:17:17.791430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791393 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.791581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791440 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.791581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791518 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791589 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791611 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.791630 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2xq\" (UniqueName: \"kubernetes.io/projected/eeda50f8-a5eb-49b3-89bd-434056f44cbe-kube-api-access-mf2xq\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892227 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892296 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892327 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2xq\" (UniqueName: \"kubernetes.io/projected/eeda50f8-a5eb-49b3-89bd-434056f44cbe-kube-api-access-mf2xq\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892394 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892442 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892475 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.892886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.892866 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-manifests\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.894560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.894542 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eeda50f8-a5eb-49b3-89bd-434056f44cbe-tmp\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.894707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.894684 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.895020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.895000 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.895102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.895055 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eeda50f8-a5eb-49b3-89bd-434056f44cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.914454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.914428 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2xq\" (UniqueName: \"kubernetes.io/projected/eeda50f8-a5eb-49b3-89bd-434056f44cbe-kube-api-access-mf2xq\") pod \"opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5\" (UID: \"eeda50f8-a5eb-49b3-89bd-434056f44cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:17.937355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:17.937329 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:18.286505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:18.286456 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5"] Oct 30 17:17:18.289888 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:18.289860 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeda50f8_a5eb_49b3_89bd_434056f44cbe.slice/crio-10420ef766cdfe078b28a657b8bdd8b775666c7016fc57e520ea6b149747d023 WatchSource:0}: Error finding container 10420ef766cdfe078b28a657b8bdd8b775666c7016fc57e520ea6b149747d023: Status 404 returned error can't find the container with id 10420ef766cdfe078b28a657b8bdd8b775666c7016fc57e520ea6b149747d023 Oct 30 17:17:18.473616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:18.473580 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" event={"ID":"eeda50f8-a5eb-49b3-89bd-434056f44cbe","Type":"ContainerStarted","Data":"10420ef766cdfe078b28a657b8bdd8b775666c7016fc57e520ea6b149747d023"} Oct 30 17:17:21.488419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:21.488330 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" event={"ID":"eeda50f8-a5eb-49b3-89bd-434056f44cbe","Type":"ContainerStarted","Data":"2fc1109a60bf77fea5df7f9b3b7f07d0c2aeb8f834280b6cee369b1e53077af0"} Oct 30 17:17:22.494732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:22.494694 4622 generic.go:356] "Generic (PLEG): container finished" podID="eeda50f8-a5eb-49b3-89bd-434056f44cbe" containerID="2fc1109a60bf77fea5df7f9b3b7f07d0c2aeb8f834280b6cee369b1e53077af0" exitCode=0 Oct 30 17:17:22.495111 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:22.494759 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" event={"ID":"eeda50f8-a5eb-49b3-89bd-434056f44cbe","Type":"ContainerDied","Data":"2fc1109a60bf77fea5df7f9b3b7f07d0c2aeb8f834280b6cee369b1e53077af0"} Oct 30 17:17:23.501762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:23.501723 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" event={"ID":"eeda50f8-a5eb-49b3-89bd-434056f44cbe","Type":"ContainerStarted","Data":"8b30e477c858c1315c50ec51f3aaf872ef236dda2afef17ca6f9c8669e8394f7"} Oct 30 17:17:23.502210 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:23.501867 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:23.586433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:23.586379 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" podStartSLOduration=2.817147045 podStartE2EDuration="6.586364002s" totalImagesPullingTime="3.314923478s" totalInitContainerRuntime="454.293479ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:23.575007543 +0000 UTC m=+271.352479497" watchObservedRunningTime="2026-10-30 17:17:23.586364002 +0000 UTC m=+271.363835956" Oct 30 17:17:24.533498 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:17:24.533457 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:17:34.509017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:34.508980 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5" Oct 30 17:17:38.256997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.256950 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2"] Oct 30 17:17:38.261339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.261318 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.284545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.284521 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xwfmd\"" Oct 30 17:17:38.284688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.284593 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 17:17:38.284688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.284604 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 17:17:38.284688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.284650 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 17:17:38.290533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.290510 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2"] Oct 30 17:17:38.359580 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.359546 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.359767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.359620 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhpz\" (UniqueName: \"kubernetes.io/projected/731b6276-2d39-422c-a21e-45978e617b98-kube-api-access-jwhpz\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.359767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.359706 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/731b6276-2d39-422c-a21e-45978e617b98-manager-config\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.359767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.359757 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.460915 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.460875 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.461112 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.460966 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhpz\" (UniqueName: \"kubernetes.io/projected/731b6276-2d39-422c-a21e-45978e617b98-kube-api-access-jwhpz\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.461112 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.461003 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/731b6276-2d39-422c-a21e-45978e617b98-manager-config\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.461246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.461218 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.461732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.461709 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/731b6276-2d39-422c-a21e-45978e617b98-manager-config\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.463403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.463382 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-metrics-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.463403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.463398 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/731b6276-2d39-422c-a21e-45978e617b98-cert\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.482683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.482656 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhpz\" (UniqueName: \"kubernetes.io/projected/731b6276-2d39-422c-a21e-45978e617b98-kube-api-access-jwhpz\") pod \"lws-controller-manager-6c446c6ff5-xpql2\" (UID: \"731b6276-2d39-422c-a21e-45978e617b98\") " pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.571506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.571478 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:38.861818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:38.861718 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2"] Oct 30 17:17:38.866215 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:38.866187 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731b6276_2d39_422c_a21e_45978e617b98.slice/crio-d5a98fb7d5575dafa1b8f5a5a36791fed6b5b6d53792299330bf7312960ab171 WatchSource:0}: Error finding container d5a98fb7d5575dafa1b8f5a5a36791fed6b5b6d53792299330bf7312960ab171: Status 404 returned error can't find the container with id d5a98fb7d5575dafa1b8f5a5a36791fed6b5b6d53792299330bf7312960ab171 Oct 30 17:17:39.569588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:39.569555 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" event={"ID":"731b6276-2d39-422c-a21e-45978e617b98","Type":"ContainerStarted","Data":"d5a98fb7d5575dafa1b8f5a5a36791fed6b5b6d53792299330bf7312960ab171"} Oct 30 17:17:41.580055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:41.580011 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" event={"ID":"731b6276-2d39-422c-a21e-45978e617b98","Type":"ContainerStarted","Data":"882f98d0e7ecaa78541d0e97def832b92c868ebf3e3322185ae741f946ab2922"} Oct 30 17:17:41.580513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:41.580133 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:41.617609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:41.617537 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" podStartSLOduration=1.825016448 podStartE2EDuration="3.617517911s" totalImagesPullingTime="1.792501463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:41.616644859 +0000 UTC m=+289.394116814" watchObservedRunningTime="2026-10-30 17:17:41.617517911 +0000 UTC m=+289.394989869" Oct 30 17:17:51.183468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.183390 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-tph5f"] Oct 30 17:17:51.186618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.186601 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.201870 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.201841 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 17:17:51.212079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.212047 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-hg744\"" Oct 30 17:17:51.213513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.213492 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-tph5f"] Oct 30 17:17:51.267570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.267534 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq77\" (UniqueName: \"kubernetes.io/projected/eee295a7-b23f-4e56-a41a-642cebd563e6-kube-api-access-slq77\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.267740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.267578 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.368506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.368471 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq77\" (UniqueName: \"kubernetes.io/projected/eee295a7-b23f-4e56-a41a-642cebd563e6-kube-api-access-slq77\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.368506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.368511 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.368755 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:17:51.368628 4622 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 17:17:51.368755 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:17:51.368695 4622 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs podName:eee295a7-b23f-4e56-a41a-642cebd563e6 nodeName:}" failed. No retries permitted until 2026-10-30 17:17:51.868680011 +0000 UTC m=+299.646151957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs") pod "odh-observability-584f99c788-tph5f" (UID: "eee295a7-b23f-4e56-a41a-642cebd563e6") : secret "odh-observability-webhook-cert" not found Oct 30 17:17:51.390679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.390645 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq77\" (UniqueName: \"kubernetes.io/projected/eee295a7-b23f-4e56-a41a-642cebd563e6-kube-api-access-slq77\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.873440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.873406 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:51.875972 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:51.875951 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eee295a7-b23f-4e56-a41a-642cebd563e6-webhook-certs\") pod \"odh-observability-584f99c788-tph5f\" (UID: \"eee295a7-b23f-4e56-a41a-642cebd563e6\") " pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:52.102145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:52.102102 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:52.393253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:52.392611 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-tph5f"] Oct 30 17:17:52.396609 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:52.396581 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee295a7_b23f_4e56_a41a_642cebd563e6.slice/crio-559a7f2995bc3c76e82f0b21223bf775f878b0f0305b1e53cb9bf4d8b30921f9 WatchSource:0}: Error finding container 559a7f2995bc3c76e82f0b21223bf775f878b0f0305b1e53cb9bf4d8b30921f9: Status 404 returned error can't find the container with id 559a7f2995bc3c76e82f0b21223bf775f878b0f0305b1e53cb9bf4d8b30921f9 Oct 30 17:17:52.587809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:52.587775 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c446c6ff5-xpql2" Oct 30 17:17:52.634440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:52.634394 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-tph5f" event={"ID":"eee295a7-b23f-4e56-a41a-642cebd563e6","Type":"ContainerStarted","Data":"559a7f2995bc3c76e82f0b21223bf775f878b0f0305b1e53cb9bf4d8b30921f9"} Oct 30 17:17:54.096079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.096046 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l"] Oct 30 17:17:54.101482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.101458 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.123069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.123037 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 17:17:54.123069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.123070 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 17:17:54.123371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.123140 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 17:17:54.128125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.128100 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l"] Oct 30 17:17:54.194470 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194438 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194487 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d1c03c-734f-42e3-98ac-d7e50594a946-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194511 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194593 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194650 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqvj\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-kube-api-access-nbqvj\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194711 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194736 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194759 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.194934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.194899 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.200041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.200010 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jcqhv\"" Oct 30 17:17:54.295836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.295801 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d1c03c-734f-42e3-98ac-d7e50594a946-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.295836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.295842 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.295864 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.295991 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqvj\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-kube-api-access-nbqvj\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296048 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296087 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296126 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296192 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296251 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296360 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296385 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296563 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296645 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d1c03c-734f-42e3-98ac-d7e50594a946-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.296820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.296799 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-data\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.298658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.298634 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.299068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.299048 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.317838 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.317807 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-istio-token\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.317995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.317907 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqvj\" (UniqueName: \"kubernetes.io/projected/43d1c03c-734f-42e3-98ac-d7e50594a946-kube-api-access-nbqvj\") pod \"data-science-gateway-data-science-gateway-class-575956cc9cpm72l\" (UID: \"43d1c03c-734f-42e3-98ac-d7e50594a946\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.418046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.417955 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:54.540232 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:17:54.540193 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:17:54.736401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.736319 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l"] Oct 30 17:17:54.739969 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:17:54.739936 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d1c03c_734f_42e3_98ac_d7e50594a946.slice/crio-657b71692c8bce26fb4f452fbdcfc44f58301c11af31ad47241de61933066755 WatchSource:0}: Error finding container 657b71692c8bce26fb4f452fbdcfc44f58301c11af31ad47241de61933066755: Status 404 returned error can't find the container with id 657b71692c8bce26fb4f452fbdcfc44f58301c11af31ad47241de61933066755 Oct 30 17:17:54.742420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:54.742400 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:17:55.651641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:55.651603 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-tph5f" event={"ID":"eee295a7-b23f-4e56-a41a-642cebd563e6","Type":"ContainerStarted","Data":"069d073091cd2fd6f6a42b41c92345e25230eff547bec096544e2b5aa582bacb"} Oct 30 17:17:55.652179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:55.651694 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:17:55.653281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:55.653231 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" event={"ID":"43d1c03c-734f-42e3-98ac-d7e50594a946","Type":"ContainerStarted","Data":"657b71692c8bce26fb4f452fbdcfc44f58301c11af31ad47241de61933066755"} Oct 30 17:17:55.690306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:55.689574 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-tph5f" podStartSLOduration=2.146766447 podStartE2EDuration="4.689553965s" totalImagesPullingTime="2.542787518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:55.688473745 +0000 UTC m=+303.465945704" watchObservedRunningTime="2026-10-30 17:17:55.689553965 +0000 UTC m=+303.467025922" Oct 30 17:17:57.384237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:57.384202 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:17:57.384661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:57.384319 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:17:57.384661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:57.384407 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:17:57.664843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:57.664756 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" event={"ID":"43d1c03c-734f-42e3-98ac-d7e50594a946","Type":"ContainerStarted","Data":"30f42500363745665a3861bd025c8803ac88f8ca9bcc322a8ef5abf3858579fe"} Oct 30 17:17:57.718409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:57.718355 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" podStartSLOduration=1.077608253 podStartE2EDuration="3.718339858s" totalImagesPullingTime="2.640731605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:57.715723671 +0000 UTC m=+305.493195626" watchObservedRunningTime="2026-10-30 17:17:57.718339858 +0000 UTC m=+305.495811889" Oct 30 17:17:58.419061 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:58.419022 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:58.423666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:58.423639 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:58.669367 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:58.669289 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:17:58.670336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:17:58.670320 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-575956cc9cpm72l" Oct 30 17:18:06.661075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:18:06.661043 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-tph5f" Oct 30 17:18:24.532717 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:18:24.532675 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:18:54.534434 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:18:54.534395 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:19:02.881695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.881659 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8tfhq"] Oct 30 17:19:02.884075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.884059 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:02.910234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.910208 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 17:19:02.918741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.918716 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8tfhq"] Oct 30 17:19:02.919419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.919403 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fh56g\"" Oct 30 17:19:02.938028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.938004 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:19:02.940300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.940282 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:02.955593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.955567 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrs5\" (UniqueName: \"kubernetes.io/projected/3e345ce8-1709-483f-845b-d4f00a9980b9-kube-api-access-hbrs5\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:02.955746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.955668 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e345ce8-1709-483f-845b-d4f00a9980b9-cert\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:02.961862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.961838 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:19:02.978148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.978121 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pf6jq"] Oct 30 17:19:02.980473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:02.980450 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.003548 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.003521 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pf6jq"] Oct 30 17:19:03.019781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.019756 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 17:19:03.019968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.019838 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 17:19:03.019968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.019940 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fct4b\"" Oct 30 17:19:03.020079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.020022 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mdwtx\"" Oct 30 17:19:03.042549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.042520 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-29dgg"] Oct 30 17:19:03.044912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.044891 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.056174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056156 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e04659f-ddde-43e6-96a2-d34c17d0cc25-tls-certs\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.056298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056187 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86769\" (UniqueName: \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.056298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056215 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrs5\" (UniqueName: \"kubernetes.io/projected/3e345ce8-1709-483f-845b-d4f00a9980b9-kube-api-access-hbrs5\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:03.056298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056282 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk987\" (UniqueName: \"kubernetes.io/projected/0e04659f-ddde-43e6-96a2-d34c17d0cc25-kube-api-access-kk987\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.056410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056315 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.056410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.056344 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e345ce8-1709-483f-845b-d4f00a9980b9-cert\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:03.058599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.058584 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e345ce8-1709-483f-845b-d4f00a9980b9-cert\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:03.067768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.067745 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-29dgg"] Oct 30 17:19:03.077907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.077879 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrs5\" (UniqueName: \"kubernetes.io/projected/3e345ce8-1709-483f-845b-d4f00a9980b9-kube-api-access-hbrs5\") pod \"kserve-controller-manager-776cc66d47-8tfhq\" (UID: \"3e345ce8-1709-483f-845b-d4f00a9980b9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:03.123778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.123750 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fm77l\"" Oct 30 17:19:03.123938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.123855 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 17:19:03.123938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.123933 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 17:19:03.157344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157245 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.157344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157321 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-metrics-certs\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.157510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157361 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-cert\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.157510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157484 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e04659f-ddde-43e6-96a2-d34c17d0cc25-tls-certs\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.157584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157519 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86769\" (UniqueName: \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.157584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157544 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk987\" (UniqueName: \"kubernetes.io/projected/0e04659f-ddde-43e6-96a2-d34c17d0cc25-kube-api-access-kk987\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.157584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.157569 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njx9w\" (UniqueName: \"kubernetes.io/projected/cf94a5f2-82ff-4636-a99c-0a36d60d180b-kube-api-access-njx9w\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.159697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.159671 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.159813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.159738 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e04659f-ddde-43e6-96a2-d34c17d0cc25-tls-certs\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.178908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.178877 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk987\" (UniqueName: \"kubernetes.io/projected/0e04659f-ddde-43e6-96a2-d34c17d0cc25-kube-api-access-kk987\") pod \"model-serving-api-75bd89fdfc-pf6jq\" (UID: \"0e04659f-ddde-43e6-96a2-d34c17d0cc25\") " pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.179028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.178947 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86769\" (UniqueName: \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\") pod \"llmisvc-controller-manager-869d5dcc5d-js49r\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.195829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.195806 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:03.252605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.252578 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:03.258321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.258112 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njx9w\" (UniqueName: \"kubernetes.io/projected/cf94a5f2-82ff-4636-a99c-0a36d60d180b-kube-api-access-njx9w\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.258321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.258153 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-metrics-certs\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.258321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.258174 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-cert\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.261754 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.261700 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-cert\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.262556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.262521 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf94a5f2-82ff-4636-a99c-0a36d60d180b-metrics-certs\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.279657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.279630 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njx9w\" (UniqueName: \"kubernetes.io/projected/cf94a5f2-82ff-4636-a99c-0a36d60d180b-kube-api-access-njx9w\") pod \"odh-model-controller-f87fcf56c-29dgg\" (UID: \"cf94a5f2-82ff-4636-a99c-0a36d60d180b\") " pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.292502 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.292475 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:03.356616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.356570 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:03.487511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.487477 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-8tfhq"] Oct 30 17:19:03.491439 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:03.491406 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e345ce8_1709_483f_845b_d4f00a9980b9.slice/crio-a4166f5954b655e74da58614eead095016d250c0ba5f58d6fc344b301be32cb9 WatchSource:0}: Error finding container a4166f5954b655e74da58614eead095016d250c0ba5f58d6fc344b301be32cb9: Status 404 returned error can't find the container with id a4166f5954b655e74da58614eead095016d250c0ba5f58d6fc344b301be32cb9 Oct 30 17:19:03.546666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.546611 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:19:03.549781 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:03.549751 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod836205fb_512c_432c_9c80_cafba4f7f732.slice/crio-615f39e6686508b1c1f0e86a0411296f5cf537ab2a0a872ec46a201bf7f74db0 WatchSource:0}: Error finding container 615f39e6686508b1c1f0e86a0411296f5cf537ab2a0a872ec46a201bf7f74db0: Status 404 returned error can't find the container with id 615f39e6686508b1c1f0e86a0411296f5cf537ab2a0a872ec46a201bf7f74db0 Oct 30 17:19:03.594746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.594713 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-pf6jq"] Oct 30 17:19:03.598132 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:03.598107 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e04659f_ddde_43e6_96a2_d34c17d0cc25.slice/crio-3dcb3a1942ae5a2142c3bb9b3e942d8545a4eb74e10f947f0000acb6d3033428 WatchSource:0}: Error finding container 3dcb3a1942ae5a2142c3bb9b3e942d8545a4eb74e10f947f0000acb6d3033428: Status 404 returned error can't find the container with id 3dcb3a1942ae5a2142c3bb9b3e942d8545a4eb74e10f947f0000acb6d3033428 Oct 30 17:19:03.649860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.649824 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-29dgg"] Oct 30 17:19:03.653854 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:03.653824 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf94a5f2_82ff_4636_a99c_0a36d60d180b.slice/crio-be1a1040188569afaf79ee56673482458ad2f58de17021559666eb09ee8033c9 WatchSource:0}: Error finding container be1a1040188569afaf79ee56673482458ad2f58de17021559666eb09ee8033c9: Status 404 returned error can't find the container with id be1a1040188569afaf79ee56673482458ad2f58de17021559666eb09ee8033c9 Oct 30 17:19:03.953412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.953378 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" event={"ID":"3e345ce8-1709-483f-845b-d4f00a9980b9","Type":"ContainerStarted","Data":"a4166f5954b655e74da58614eead095016d250c0ba5f58d6fc344b301be32cb9"} Oct 30 17:19:03.954860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.954820 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" event={"ID":"cf94a5f2-82ff-4636-a99c-0a36d60d180b","Type":"ContainerStarted","Data":"be1a1040188569afaf79ee56673482458ad2f58de17021559666eb09ee8033c9"} Oct 30 17:19:03.955924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.955900 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" event={"ID":"836205fb-512c-432c-9c80-cafba4f7f732","Type":"ContainerStarted","Data":"615f39e6686508b1c1f0e86a0411296f5cf537ab2a0a872ec46a201bf7f74db0"} Oct 30 17:19:03.957351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:03.957328 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" event={"ID":"0e04659f-ddde-43e6-96a2-d34c17d0cc25","Type":"ContainerStarted","Data":"3dcb3a1942ae5a2142c3bb9b3e942d8545a4eb74e10f947f0000acb6d3033428"} Oct 30 17:19:06.982288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:06.982224 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" event={"ID":"0e04659f-ddde-43e6-96a2-d34c17d0cc25","Type":"ContainerStarted","Data":"d0c9af9262f8f019a2bdd982b40e118f6bdae6db8ef8c9707e9d72abb4984f90"} Oct 30 17:19:06.982698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:06.982437 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:07.020170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.020065 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" podStartSLOduration=1.859281217 podStartE2EDuration="5.020047008s" totalImagesPullingTime="3.160765791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:07.019122834 +0000 UTC m=+374.796594790" watchObservedRunningTime="2026-10-30 17:19:07.020047008 +0000 UTC m=+374.797518965" Oct 30 17:19:07.988974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.988934 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" event={"ID":"3e345ce8-1709-483f-845b-d4f00a9980b9","Type":"ContainerStarted","Data":"0c20117019ccf10abad43bdfd5bdec760cacff96601ebfd62e1debfb0c4f7b2c"} Oct 30 17:19:07.989431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.989037 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:07.990573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.990548 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" event={"ID":"cf94a5f2-82ff-4636-a99c-0a36d60d180b","Type":"ContainerStarted","Data":"24e589f9131491e2cdecca805df039cd78d669ec5cbd7e4992d4becd9a40adaa"} Oct 30 17:19:07.990719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.990599 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:07.992124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.992096 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" event={"ID":"836205fb-512c-432c-9c80-cafba4f7f732","Type":"ContainerStarted","Data":"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66"} Oct 30 17:19:07.992226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:07.992159 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:08.026162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:08.026115 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" podStartSLOduration=1.687479754 podStartE2EDuration="6.026102339s" totalImagesPullingTime="4.338622585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:08.024660772 +0000 UTC m=+375.802132726" watchObservedRunningTime="2026-10-30 17:19:08.026102339 +0000 UTC m=+375.803574303" Oct 30 17:19:08.072938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:08.072838 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" podStartSLOduration=0.909552641 podStartE2EDuration="5.072822516s" totalImagesPullingTime="4.163269875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:08.071823726 +0000 UTC m=+375.849295681" watchObservedRunningTime="2026-10-30 17:19:08.072822516 +0000 UTC m=+375.850294471" Oct 30 17:19:08.120076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:08.119954 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" podStartSLOduration=1.852745834 podStartE2EDuration="6.119933907s" totalImagesPullingTime="4.267188073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:08.118478338 +0000 UTC m=+375.895950289" watchObservedRunningTime="2026-10-30 17:19:08.119933907 +0000 UTC m=+375.897405862" Oct 30 17:19:17.996248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:17.996221 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-pf6jq" Oct 30 17:19:18.998151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:18.998118 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-29dgg" Oct 30 17:19:24.533173 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:19:24.533096 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:19:38.999148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:38.999115 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:19:39.001946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:39.001926 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-8tfhq" Oct 30 17:19:52.380466 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:52.380438 4622 scope.go:122] "RemoveContainer" containerID="18c9c6f7a378fdba223c8d205d714c5cd37ba4690b2c664be362e567434a4fbc" Oct 30 17:19:53.325867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.325832 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8"] Oct 30 17:19:53.329354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.329333 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.349586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.349561 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-mw5ss\"" Oct 30 17:19:53.360024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.359996 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 17:19:53.360208 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.360161 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 17:19:53.361918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.361890 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8"] Oct 30 17:19:53.416759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.416725 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.417174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.416786 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.417174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.416915 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwt5d\" (UniqueName: \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.418417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.418392 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth"] Oct 30 17:19:53.421078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.421059 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.436283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.436242 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-jt6r5\"" Oct 30 17:19:53.441547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.441521 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth"] Oct 30 17:19:53.514830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.514800 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8"] Oct 30 17:19:53.517533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517513 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.517661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517630 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.517706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517679 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.517757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517725 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwt5d\" (UniqueName: \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.517849 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517826 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8jh\" (UniqueName: \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.517930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517910 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.517985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.517964 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.518354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.518334 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.518354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.518347 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.529840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.529818 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2hb6g\"" Oct 30 17:19:53.537547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.537523 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8"] Oct 30 17:19:53.539083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.539055 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwt5d\" (UniqueName: \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.617962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.617882 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb"] Oct 30 17:19:53.619257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619230 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.619385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619299 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlh7\" (UniqueName: \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.619449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619424 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.619506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619480 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.619559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619514 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.619615 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619590 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8jh\" (UniqueName: \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.619831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619808 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.619917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.619817 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.620825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.620811 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.633452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.633431 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-sc4kk\"" Oct 30 17:19:53.640789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.640759 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8jh\" (UniqueName: \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.642610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.642595 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:53.647063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.647029 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb"] Oct 30 17:19:53.720874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.720840 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.721054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.720904 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rc4\" (UniqueName: \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.721054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.720950 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.722057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.721292 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.722057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.721450 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.722057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.721509 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.722057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.721530 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlh7\" (UniqueName: \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.722057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.722044 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.733753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.733721 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:53.742910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.742891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlh7\" (UniqueName: \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.823022 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.822989 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.823228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.823103 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rc4\" (UniqueName: \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.823228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.823150 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.823429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.823409 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.823515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.823497 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.830894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.830876 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:53.844639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.844610 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rc4\" (UniqueName: \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.933509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.933472 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:53.986596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:53.985338 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8"] Oct 30 17:19:53.989794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:53.989748 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a99129_810f_4e29_be93_57cf3eb1410c.slice/crio-abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821 WatchSource:0}: Error finding container abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821: Status 404 returned error can't find the container with id abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821 Oct 30 17:19:54.037036 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.037010 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth"] Oct 30 17:19:54.039916 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:54.039891 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4da8f51_b61e_4fb7_8ddd_5fa6aa69e988.slice/crio-d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4 WatchSource:0}: Error finding container d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4: Status 404 returned error can't find the container with id d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4 Oct 30 17:19:54.124871 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.124799 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8"] Oct 30 17:19:54.147275 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:54.147226 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65f5d19_8fc0_4f99_b62f_2b97269c5928.slice/crio-e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7 WatchSource:0}: Error finding container e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7: Status 404 returned error can't find the container with id e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7 Oct 30 17:19:54.209042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.209004 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" event={"ID":"a65f5d19-8fc0-4f99-b62f-2b97269c5928","Type":"ContainerStarted","Data":"e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7"} Oct 30 17:19:54.210788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.210762 4622 generic.go:356] "Generic (PLEG): container finished" podID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerID="ab71236ea514ae7423a1b17a8f593910794a77108fb8aedfcc4668964cc02c49" exitCode=0 Oct 30 17:19:54.210988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.210859 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" event={"ID":"c9a99129-810f-4e29-be93-57cf3eb1410c","Type":"ContainerDied","Data":"ab71236ea514ae7423a1b17a8f593910794a77108fb8aedfcc4668964cc02c49"} Oct 30 17:19:54.210988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.210918 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" event={"ID":"c9a99129-810f-4e29-be93-57cf3eb1410c","Type":"ContainerStarted","Data":"abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821"} Oct 30 17:19:54.212569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.212549 4622 generic.go:356] "Generic (PLEG): container finished" podID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerID="12b72c5dc8f8627e2c5ab2eb0f7b19664a9f8ef06a59fe766920f463a739a460" exitCode=0 Oct 30 17:19:54.212680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.212662 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" event={"ID":"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988","Type":"ContainerDied","Data":"12b72c5dc8f8627e2c5ab2eb0f7b19664a9f8ef06a59fe766920f463a739a460"} Oct 30 17:19:54.212730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.212704 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" event={"ID":"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988","Type":"ContainerStarted","Data":"d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4"} Oct 30 17:19:54.228153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:54.228123 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb"] Oct 30 17:19:54.238389 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:19:54.238362 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfec58f9_ff62_4bda_960c_30fe8869099f.slice/crio-5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163 WatchSource:0}: Error finding container 5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163: Status 404 returned error can't find the container with id 5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163 Oct 30 17:19:54.534031 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:19:54.533995 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:19:55.219873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.219839 4622 generic.go:356] "Generic (PLEG): container finished" podID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerID="cb19af8d06cb74d73fc4d0d366ea5c9c88b5e84ec5639cf8eaa4a0a4ba56e6aa" exitCode=0 Oct 30 17:19:55.220054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.219915 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" event={"ID":"dfec58f9-ff62-4bda-960c-30fe8869099f","Type":"ContainerDied","Data":"cb19af8d06cb74d73fc4d0d366ea5c9c88b5e84ec5639cf8eaa4a0a4ba56e6aa"} Oct 30 17:19:55.220054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.220001 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" event={"ID":"dfec58f9-ff62-4bda-960c-30fe8869099f","Type":"ContainerStarted","Data":"5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163"} Oct 30 17:19:55.221698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.221672 4622 generic.go:356] "Generic (PLEG): container finished" podID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerID="e7ae38dca59833e0f8b19381552b115bd11ae8ff88c3db3541e0940b2c89190e" exitCode=0 Oct 30 17:19:55.221833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.221758 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" event={"ID":"a65f5d19-8fc0-4f99-b62f-2b97269c5928","Type":"ContainerDied","Data":"e7ae38dca59833e0f8b19381552b115bd11ae8ff88c3db3541e0940b2c89190e"} Oct 30 17:19:55.223848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.223814 4622 generic.go:356] "Generic (PLEG): container finished" podID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerID="a39b1feda72bce197fef7c56083bb07d7c25868c5c9358a01d41fcdfcb191cda" exitCode=0 Oct 30 17:19:55.223958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:55.223858 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" event={"ID":"c9a99129-810f-4e29-be93-57cf3eb1410c","Type":"ContainerDied","Data":"a39b1feda72bce197fef7c56083bb07d7c25868c5c9358a01d41fcdfcb191cda"} Oct 30 17:19:56.233581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.233541 4622 generic.go:356] "Generic (PLEG): container finished" podID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerID="0724d118364ce81b7ffec0b29981eb9e18ae3831481bcc236beb53a26193a266" exitCode=0 Oct 30 17:19:56.233962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.233636 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" event={"ID":"a65f5d19-8fc0-4f99-b62f-2b97269c5928","Type":"ContainerDied","Data":"0724d118364ce81b7ffec0b29981eb9e18ae3831481bcc236beb53a26193a266"} Oct 30 17:19:56.236326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.236303 4622 generic.go:356] "Generic (PLEG): container finished" podID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerID="6e24ae8957d3cb11eec4ac03748eff84157774f37f5fbc8333eb20a5a1eded96" exitCode=0 Oct 30 17:19:56.236432 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.236382 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" event={"ID":"c9a99129-810f-4e29-be93-57cf3eb1410c","Type":"ContainerDied","Data":"6e24ae8957d3cb11eec4ac03748eff84157774f37f5fbc8333eb20a5a1eded96"} Oct 30 17:19:56.238419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.238393 4622 generic.go:356] "Generic (PLEG): container finished" podID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerID="367442d7b4b1cbeb5700f78f017e9eddd8a3dc37a77e900e58255ca99a00cd6c" exitCode=0 Oct 30 17:19:56.238539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:56.238499 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" event={"ID":"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988","Type":"ContainerDied","Data":"367442d7b4b1cbeb5700f78f017e9eddd8a3dc37a77e900e58255ca99a00cd6c"} Oct 30 17:19:57.245139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.245107 4622 generic.go:356] "Generic (PLEG): container finished" podID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerID="5ce3d302f31346321f8c123a7952a10f8c05eab6dbb4e852da8be7da9f2dea70" exitCode=0 Oct 30 17:19:57.245611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.245195 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" event={"ID":"a65f5d19-8fc0-4f99-b62f-2b97269c5928","Type":"ContainerDied","Data":"5ce3d302f31346321f8c123a7952a10f8c05eab6dbb4e852da8be7da9f2dea70"} Oct 30 17:19:57.247141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.247120 4622 generic.go:356] "Generic (PLEG): container finished" podID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerID="09c386c4ce7feae3c642a43698bcf8fccf09698ffba39fd3979c11d55f7c8f1d" exitCode=0 Oct 30 17:19:57.247253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.247195 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" event={"ID":"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988","Type":"ContainerDied","Data":"09c386c4ce7feae3c642a43698bcf8fccf09698ffba39fd3979c11d55f7c8f1d"} Oct 30 17:19:57.248830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.248812 4622 generic.go:356] "Generic (PLEG): container finished" podID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerID="a9d983bce3a93da9cdfe9854d6437c346fa6d9b3afc8d91f8ae5e0c7ad692c58" exitCode=0 Oct 30 17:19:57.248923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.248887 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" event={"ID":"dfec58f9-ff62-4bda-960c-30fe8869099f","Type":"ContainerDied","Data":"a9d983bce3a93da9cdfe9854d6437c346fa6d9b3afc8d91f8ae5e0c7ad692c58"} Oct 30 17:19:57.386237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.386213 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:57.454945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.454910 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\") pod \"c9a99129-810f-4e29-be93-57cf3eb1410c\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " Oct 30 17:19:57.455137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.455004 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\" (UniqueName: \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\") pod \"c9a99129-810f-4e29-be93-57cf3eb1410c\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " Oct 30 17:19:57.455137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.455044 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\") pod \"c9a99129-810f-4e29-be93-57cf3eb1410c\" (UID: \"c9a99129-810f-4e29-be93-57cf3eb1410c\") " Oct 30 17:19:57.455664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.455640 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle" pod "c9a99129-810f-4e29-be93-57cf3eb1410c" (UID: "c9a99129-810f-4e29-be93-57cf3eb1410c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:57.457137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.457113 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d" pod "c9a99129-810f-4e29-be93-57cf3eb1410c" (UID: "c9a99129-810f-4e29-be93-57cf3eb1410c"). InnerVolumeSpecName "kube-api-access-vwt5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:57.461205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.461179 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util" pod "c9a99129-810f-4e29-be93-57cf3eb1410c" (UID: "c9a99129-810f-4e29-be93-57cf3eb1410c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:57.556513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.556471 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwt5d\" (UniqueName: \"kubernetes.io/projected/c9a99129-810f-4e29-be93-57cf3eb1410c-kube-api-access-vwt5d\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:57.556513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.556509 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:57.556513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:57.556519 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9a99129-810f-4e29-be93-57cf3eb1410c-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.255817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.255782 4622 generic.go:356] "Generic (PLEG): container finished" podID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerID="96b87225c048c972d0daf58fa6648c6615e104060324fbc599dafa9293325a4f" exitCode=0 Oct 30 17:19:58.256254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.255874 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" event={"ID":"dfec58f9-ff62-4bda-960c-30fe8869099f","Type":"ContainerDied","Data":"96b87225c048c972d0daf58fa6648c6615e104060324fbc599dafa9293325a4f"} Oct 30 17:19:58.257776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.257757 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" Oct 30 17:19:58.257893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.257790 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crtzz8" event={"ID":"c9a99129-810f-4e29-be93-57cf3eb1410c","Type":"ContainerDied","Data":"abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821"} Oct 30 17:19:58.257893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.257826 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abb4ecf42a57d4458b48e45dc7fa0245b1b19241391e15970e9efb5b6396a821" Oct 30 17:19:58.408190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.408164 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:58.411209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.411192 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:58.463874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.463846 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\") pod \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " Oct 30 17:19:58.464031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.463912 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\") pod \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " Oct 30 17:19:58.464031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.463937 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\") pod \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " Oct 30 17:19:58.464031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.463962 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\" (UniqueName: \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\") pod \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " Oct 30 17:19:58.464031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.463987 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\") pod \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\" (UID: \"a65f5d19-8fc0-4f99-b62f-2b97269c5928\") " Oct 30 17:19:58.464031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.464017 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\" (UniqueName: \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\") pod \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\" (UID: \"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988\") " Oct 30 17:19:58.464812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.464770 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle" pod "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" (UID: "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:58.464942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.464809 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle" pod "a65f5d19-8fc0-4f99-b62f-2b97269c5928" (UID: "a65f5d19-8fc0-4f99-b62f-2b97269c5928"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:58.466304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.466276 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7" pod "a65f5d19-8fc0-4f99-b62f-2b97269c5928" (UID: "a65f5d19-8fc0-4f99-b62f-2b97269c5928"). InnerVolumeSpecName "kube-api-access-hhlh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:58.466393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.466306 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh" pod "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" (UID: "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988"). InnerVolumeSpecName "kube-api-access-lh8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:58.472687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.472643 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util" pod "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" (UID: "e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:58.472960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.472936 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util" pod "a65f5d19-8fc0-4f99-b62f-2b97269c5928" (UID: "a65f5d19-8fc0-4f99-b62f-2b97269c5928"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:58.565673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565639 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.565673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565670 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.565673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565680 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.565895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565689 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlh7\" (UniqueName: \"kubernetes.io/projected/a65f5d19-8fc0-4f99-b62f-2b97269c5928-kube-api-access-hhlh7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.565895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565700 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65f5d19-8fc0-4f99-b62f-2b97269c5928-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:58.565895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:58.565710 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh8jh\" (UniqueName: \"kubernetes.io/projected/e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988-kube-api-access-lh8jh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:59.263923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.263889 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" Oct 30 17:19:59.264388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.263889 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dlzg8" event={"ID":"a65f5d19-8fc0-4f99-b62f-2b97269c5928","Type":"ContainerDied","Data":"e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7"} Oct 30 17:19:59.264388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.264024 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90bb75828ff6117ca57987d19d997790967cadc7eaeaa8eac1d9a52a546afc7" Oct 30 17:19:59.265876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.265853 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" Oct 30 17:19:59.265992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.265881 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aslfth" event={"ID":"e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988","Type":"ContainerDied","Data":"d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4"} Oct 30 17:19:59.265992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.265920 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57ad2b82df02d0fcc6da279b0f670eed1b2ec8ab789704481770785e37a4eb4" Oct 30 17:19:59.395221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.395198 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:19:59.474840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.474798 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\") pod \"dfec58f9-ff62-4bda-960c-30fe8869099f\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " Oct 30 17:19:59.475076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.474934 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\" (UniqueName: \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\") pod \"dfec58f9-ff62-4bda-960c-30fe8869099f\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " Oct 30 17:19:59.475076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.474994 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\") pod \"dfec58f9-ff62-4bda-960c-30fe8869099f\" (UID: \"dfec58f9-ff62-4bda-960c-30fe8869099f\") " Oct 30 17:19:59.475463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.475440 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle" pod "dfec58f9-ff62-4bda-960c-30fe8869099f" (UID: "dfec58f9-ff62-4bda-960c-30fe8869099f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:59.476986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.476965 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4" pod "dfec58f9-ff62-4bda-960c-30fe8869099f" (UID: "dfec58f9-ff62-4bda-960c-30fe8869099f"). InnerVolumeSpecName "kube-api-access-26rc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:59.497473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.497435 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util" pod "dfec58f9-ff62-4bda-960c-30fe8869099f" (UID: "dfec58f9-ff62-4bda-960c-30fe8869099f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:59.576606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.576560 4622 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-bundle\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:59.576606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.576606 4622 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfec58f9-ff62-4bda-960c-30fe8869099f-util\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:19:59.576789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:19:59.576617 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rc4\" (UniqueName: \"kubernetes.io/projected/dfec58f9-ff62-4bda-960c-30fe8869099f-kube-api-access-26rc4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:20:00.272933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:00.272902 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" Oct 30 17:20:00.272933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:00.272906 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62kjpb" event={"ID":"dfec58f9-ff62-4bda-960c-30fe8869099f","Type":"ContainerDied","Data":"5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163"} Oct 30 17:20:00.273394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:00.272960 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4d92519774371cb89e8280ea08b93be87e71ede473dad4a6151aec4677e163" Oct 30 17:20:06.979475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979432 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7"] Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979813 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="util" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979831 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="util" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979843 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="extract" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979850 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="extract" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979860 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="extract" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979869 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="extract" Oct 30 17:20:06.979887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979886 4622 cpu_manager.go:414] "containerMap: removing container" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979894 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979905 4622 cpu_manager.go:414] "containerMap: removing container" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979912 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979930 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979936 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979944 4622 cpu_manager.go:414] "containerMap: removing container" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979950 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979968 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979973 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979982 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.979988 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980011 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980017 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980031 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980037 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="util" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980046 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980052 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="pull" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980168 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65f5d19-8fc0-4f99-b62f-2b97269c5928" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980183 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a99129-810f-4e29-be93-57cf3eb1410c" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980190 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfec58f9-ff62-4bda-960c-30fe8869099f" containerName="extract" Oct 30 17:20:06.980194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.980199 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4da8f51-b61e-4fb7-8ddd-5fa6aa69e988" containerName="extract" Oct 30 17:20:06.986709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:06.983825 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:07.009977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.009949 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kvlrk\"" Oct 30 17:20:07.009977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.009968 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 17:20:07.010177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.009947 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 17:20:07.010177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.009988 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 17:20:07.013819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.013793 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7"] Oct 30 17:20:07.144821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.144789 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8b8\" (UniqueName: \"kubernetes.io/projected/05be6fab-f98b-42cf-9c25-9b4605b47504-kube-api-access-tn8b8\") pod \"dns-operator-controller-manager-65bfd79598-vbpd7\" (UID: \"05be6fab-f98b-42cf-9c25-9b4605b47504\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:07.245402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.245313 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8b8\" (UniqueName: \"kubernetes.io/projected/05be6fab-f98b-42cf-9c25-9b4605b47504-kube-api-access-tn8b8\") pod \"dns-operator-controller-manager-65bfd79598-vbpd7\" (UID: \"05be6fab-f98b-42cf-9c25-9b4605b47504\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:07.267603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.267576 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8b8\" (UniqueName: \"kubernetes.io/projected/05be6fab-f98b-42cf-9c25-9b4605b47504-kube-api-access-tn8b8\") pod \"dns-operator-controller-manager-65bfd79598-vbpd7\" (UID: \"05be6fab-f98b-42cf-9c25-9b4605b47504\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:07.297224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.297197 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:07.592705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:07.592676 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7"] Oct 30 17:20:08.312863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:08.312826 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" event={"ID":"05be6fab-f98b-42cf-9c25-9b4605b47504","Type":"ContainerStarted","Data":"bce3619d5d760e39032ffa83ad5a667c03a3bc3880531b928787be1b3c8d5470"} Oct 30 17:20:10.325714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:10.325681 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" event={"ID":"05be6fab-f98b-42cf-9c25-9b4605b47504","Type":"ContainerStarted","Data":"bebe398a3aa82371a3beb222393a2974323562aa8760804fbe77d7960fc1c2d9"} Oct 30 17:20:10.326145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:10.325801 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:10.378071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:10.378004 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" podStartSLOduration=1.8423892670000002 podStartE2EDuration="4.377962234s" totalImagesPullingTime="2.535572967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:10.377145749 +0000 UTC m=+438.154617704" watchObservedRunningTime="2026-10-30 17:20:10.377962234 +0000 UTC m=+438.155434188" Oct 30 17:20:13.661782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.661752 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm"] Oct 30 17:20:13.665338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.665321 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:13.687998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.687971 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tjwtz\"" Oct 30 17:20:13.698913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.698881 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm"] Oct 30 17:20:13.803250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.803215 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5mpq\" (UniqueName: \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\") pod \"limitador-operator-controller-manager-856dd584c5-7llsm\" (UID: \"ddce1cec-eb12-486f-ae39-3f6278c498c4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:13.904000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.903965 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5mpq\" (UniqueName: \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\") pod \"limitador-operator-controller-manager-856dd584c5-7llsm\" (UID: \"ddce1cec-eb12-486f-ae39-3f6278c498c4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:13.925482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.925416 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5mpq\" (UniqueName: \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\") pod \"limitador-operator-controller-manager-856dd584c5-7llsm\" (UID: \"ddce1cec-eb12-486f-ae39-3f6278c498c4\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:13.977868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:13.977839 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:14.271960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:14.271874 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm"] Oct 30 17:20:14.275043 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:20:14.274998 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddce1cec_eb12_486f_ae39_3f6278c498c4.slice/crio-e0ff8bcc16fc6fde8193c4c87cc18615e7c251a0fa1d57db63051a37565b4dc1 WatchSource:0}: Error finding container e0ff8bcc16fc6fde8193c4c87cc18615e7c251a0fa1d57db63051a37565b4dc1: Status 404 returned error can't find the container with id e0ff8bcc16fc6fde8193c4c87cc18615e7c251a0fa1d57db63051a37565b4dc1 Oct 30 17:20:14.346464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:14.346435 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" event={"ID":"ddce1cec-eb12-486f-ae39-3f6278c498c4","Type":"ContainerStarted","Data":"e0ff8bcc16fc6fde8193c4c87cc18615e7c251a0fa1d57db63051a37565b4dc1"} Oct 30 17:20:16.358862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:16.358824 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" event={"ID":"ddce1cec-eb12-486f-ae39-3f6278c498c4","Type":"ContainerStarted","Data":"52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e"} Oct 30 17:20:16.359316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:16.358944 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:16.396454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:16.396399 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" podStartSLOduration=1.6949465259999998 podStartE2EDuration="3.396385156s" totalImagesPullingTime="1.70143863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:16.395363171 +0000 UTC m=+444.172835127" watchObservedRunningTime="2026-10-30 17:20:16.396385156 +0000 UTC m=+444.173857110" Oct 30 17:20:21.333475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:21.333446 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-vbpd7" Oct 30 17:20:24.533287 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:24.533229 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:20:27.366010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:27.365977 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:41.417647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.417615 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm"] Oct 30 17:20:41.418205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.417828 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" containerName="manager" containerID="cri-o://52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e" gracePeriod=2 Oct 30 17:20:41.429790 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:41.429667 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7llsm\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:41.430713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.430687 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm"] Oct 30 17:20:41.445595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.445029 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8"] Oct 30 17:20:41.445595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.445450 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" containerName="manager" Oct 30 17:20:41.445595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.445462 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" containerName="manager" Oct 30 17:20:41.445595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.445548 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" containerName="manager" Oct 30 17:20:41.448535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.448517 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:41.471619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.471591 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8"] Oct 30 17:20:41.499148 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:41.499113 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7llsm\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:41.534695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.534675 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7hgr\" (UniqueName: \"kubernetes.io/projected/f7fa3d37-0764-48d9-8936-3bdbafd810e5-kube-api-access-h7hgr\") pod \"limitador-operator-controller-manager-856dd584c5-292f8\" (UID: \"f7fa3d37-0764-48d9-8936-3bdbafd810e5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:41.636001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.635955 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7hgr\" (UniqueName: \"kubernetes.io/projected/f7fa3d37-0764-48d9-8936-3bdbafd810e5-kube-api-access-h7hgr\") pod \"limitador-operator-controller-manager-856dd584c5-292f8\" (UID: \"f7fa3d37-0764-48d9-8936-3bdbafd810e5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:41.662054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.662031 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7hgr\" (UniqueName: \"kubernetes.io/projected/f7fa3d37-0764-48d9-8936-3bdbafd810e5-kube-api-access-h7hgr\") pod \"limitador-operator-controller-manager-856dd584c5-292f8\" (UID: \"f7fa3d37-0764-48d9-8936-3bdbafd810e5\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:41.665016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.664995 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:41.676443 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:41.676392 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7llsm\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:41.736969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.736939 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\" (UniqueName: \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\") pod \"ddce1cec-eb12-486f-ae39-3f6278c498c4\" (UID: \"ddce1cec-eb12-486f-ae39-3f6278c498c4\") " Oct 30 17:20:41.738999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.738973 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq" pod "ddce1cec-eb12-486f-ae39-3f6278c498c4" (UID: "ddce1cec-eb12-486f-ae39-3f6278c498c4"). InnerVolumeSpecName "kube-api-access-j5mpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:41.799960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.799938 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:41.838637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:41.838608 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5mpq\" (UniqueName: \"kubernetes.io/projected/ddce1cec-eb12-486f-ae39-3f6278c498c4-kube-api-access-j5mpq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:20:42.101953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.101915 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8"] Oct 30 17:20:42.105662 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:20:42.105637 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa3d37_0764_48d9_8936_3bdbafd810e5.slice/crio-4eccbd85df565737fe27e0d8bf5fe7193b46285f465aa9ed140dc68b96a4d3e4 WatchSource:0}: Error finding container 4eccbd85df565737fe27e0d8bf5fe7193b46285f465aa9ed140dc68b96a4d3e4: Status 404 returned error can't find the container with id 4eccbd85df565737fe27e0d8bf5fe7193b46285f465aa9ed140dc68b96a4d3e4 Oct 30 17:20:42.372154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.372067 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" path="/var/lib/kubelet/pods/ddce1cec-eb12-486f-ae39-3f6278c498c4/volumes" Oct 30 17:20:42.380373 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:42.380335 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-7llsm\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ddce1cec-eb12-486f-ae39-3f6278c498c4" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:42.485885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.485845 4622 generic.go:356] "Generic (PLEG): container finished" podID="ddce1cec-eb12-486f-ae39-3f6278c498c4" containerID="52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e" exitCode=0 Oct 30 17:20:42.486378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.485914 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-7llsm" Oct 30 17:20:42.486378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.485947 4622 scope.go:122] "RemoveContainer" containerID="52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e" Oct 30 17:20:42.492180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.492142 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" event={"ID":"f7fa3d37-0764-48d9-8936-3bdbafd810e5","Type":"ContainerStarted","Data":"b3001d190652d17e9d08f51f2cf7f71f49d05e9e3aff548e9766cc272fec813b"} Oct 30 17:20:42.492306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.492253 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:42.492364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.492309 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" event={"ID":"f7fa3d37-0764-48d9-8936-3bdbafd810e5","Type":"ContainerStarted","Data":"4eccbd85df565737fe27e0d8bf5fe7193b46285f465aa9ed140dc68b96a4d3e4"} Oct 30 17:20:42.499402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.499386 4622 scope.go:122] "RemoveContainer" containerID="52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e" Oct 30 17:20:42.499682 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:42.499663 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e\": container with ID starting with 52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e not found: ID does not exist" containerID="52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e" Oct 30 17:20:42.499733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.499692 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e"} err="failed to get container status \"52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e\": rpc error: code = NotFound desc = could not find container \"52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e\": container with ID starting with 52e44443d205a392f3b06d0b85833376764ec6315e89f870ad475dbd061e2b3e not found: ID does not exist" Oct 30 17:20:42.532947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:42.532908 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" podStartSLOduration=1.532896509 podStartE2EDuration="1.532896509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:42.531009461 +0000 UTC m=+470.308481415" watchObservedRunningTime="2026-10-30 17:20:42.532896509 +0000 UTC m=+470.310368464" Oct 30 17:20:53.500063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:20:53.500030 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-292f8" Oct 30 17:20:54.533490 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:20:54.533455 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:21:10.821316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.821281 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:21:10.824794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.824775 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.852856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.852826 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:21:10.853929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.853909 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mccnk\"" Oct 30 17:21:10.889732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889702 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.889880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889734 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfg8\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.889880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889754 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.889957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889878 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.889957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889912 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.889957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889936 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.890045 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.889983 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.890045 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.890023 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.890108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.890071 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990491 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990542 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990605 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990633 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfg8\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990660 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990703 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990854 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990923 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.990985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.990949 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.991200 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.991178 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.991272 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.991232 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.991593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.991454 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.991593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.991546 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.991748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.991711 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.992877 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.992856 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:10.993317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:10.993295 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:11.011738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.011710 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:11.011938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.011913 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfg8\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\") pod \"maas-default-gateway-openshift-default-7df854bf67-v6484\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:11.138533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.138456 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:11.448752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.448673 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:21:11.451696 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:21:11.451667 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586b4908_f577_43ab_9d81_5c163ca66412.slice/crio-9c25eb67ba350cb5c8c775808bcaa124c8fd9af023ea3c25c5b5f8a8e7ac0b49 WatchSource:0}: Error finding container 9c25eb67ba350cb5c8c775808bcaa124c8fd9af023ea3c25c5b5f8a8e7ac0b49: Status 404 returned error can't find the container with id 9c25eb67ba350cb5c8c775808bcaa124c8fd9af023ea3c25c5b5f8a8e7ac0b49 Oct 30 17:21:11.453648 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.453619 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:21:11.453737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.453679 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:21:11.453737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.453710 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:21:11.637338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.637302 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerStarted","Data":"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c"} Oct 30 17:21:11.637523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.637382 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerStarted","Data":"9c25eb67ba350cb5c8c775808bcaa124c8fd9af023ea3c25c5b5f8a8e7ac0b49"} Oct 30 17:21:11.689366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:11.689317 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podStartSLOduration=1.689304206 podStartE2EDuration="1.689304206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:11.678547579 +0000 UTC m=+499.456019533" watchObservedRunningTime="2026-10-30 17:21:11.689304206 +0000 UTC m=+499.466776160" Oct 30 17:21:12.139128 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:12.139095 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:12.144349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:12.144326 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:12.642112 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:12.642079 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:12.643146 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:12.643128 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:21:24.532006 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:21:24.531971 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:21:27.567649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.567620 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:21:27.571166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.571147 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:21:27.589716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.589690 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:21:27.597817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.597796 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wv7nq\"" Oct 30 17:21:27.615471 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.615450 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsnr\" (UniqueName: \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\") pod \"authorino-56c6c9ddd5-df2rd\" (UID: \"325ac348-930d-4a62-9814-0ae65ea6940d\") " pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:21:27.716063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.716037 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsnr\" (UniqueName: \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\") pod \"authorino-56c6c9ddd5-df2rd\" (UID: \"325ac348-930d-4a62-9814-0ae65ea6940d\") " pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:21:27.737734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.737700 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsnr\" (UniqueName: \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\") pod \"authorino-56c6c9ddd5-df2rd\" (UID: \"325ac348-930d-4a62-9814-0ae65ea6940d\") " pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:21:27.883244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:27.883168 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:21:28.173364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:28.173293 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:21:28.176563 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:21:28.176536 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod325ac348_930d_4a62_9814_0ae65ea6940d.slice/crio-c5c80f8cc2e59faaca6fbc5f356305a5a9a3a73b4c48f304dd4c22e156166e10 WatchSource:0}: Error finding container c5c80f8cc2e59faaca6fbc5f356305a5a9a3a73b4c48f304dd4c22e156166e10: Status 404 returned error can't find the container with id c5c80f8cc2e59faaca6fbc5f356305a5a9a3a73b4c48f304dd4c22e156166e10 Oct 30 17:21:28.725346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:28.725308 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" event={"ID":"325ac348-930d-4a62-9814-0ae65ea6940d","Type":"ContainerStarted","Data":"c5c80f8cc2e59faaca6fbc5f356305a5a9a3a73b4c48f304dd4c22e156166e10"} Oct 30 17:21:29.731651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:29.731610 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" event={"ID":"325ac348-930d-4a62-9814-0ae65ea6940d","Type":"ContainerStarted","Data":"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2"} Oct 30 17:21:29.767243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:29.767192 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" podStartSLOduration=1.359737174 podStartE2EDuration="2.767180074s" totalImagesPullingTime="1.4074429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:29.765910606 +0000 UTC m=+517.543382561" watchObservedRunningTime="2026-10-30 17:21:29.767180074 +0000 UTC m=+517.544652030" Oct 30 17:21:54.532878 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:21:54.532842 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:21:58.466331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.466297 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxgcv"] Oct 30 17:21:58.469828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.469808 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.486812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.486791 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxgcv"] Oct 30 17:21:58.570436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.570397 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptx4g\" (UniqueName: \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\") pod \"authorino-6f6944bcdd-pxgcv\" (UID: \"2742690d-35ea-4b48-8631-cff4b7ab7cd3\") " pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.671068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.671033 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptx4g\" (UniqueName: \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\") pod \"authorino-6f6944bcdd-pxgcv\" (UID: \"2742690d-35ea-4b48-8631-cff4b7ab7cd3\") " pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.682459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.682428 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxgcv"] Oct 30 17:21:58.682660 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:21:58.682642 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ptx4g], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" podUID="2742690d-35ea-4b48-8631-cff4b7ab7cd3" Oct 30 17:21:58.695124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.695095 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptx4g\" (UniqueName: \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\") pod \"authorino-6f6944bcdd-pxgcv\" (UID: \"2742690d-35ea-4b48-8631-cff4b7ab7cd3\") " pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.710273 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.710237 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:21:58.713798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.713781 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.731826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.731803 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 17:21:58.734165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.734144 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:21:58.771921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.771891 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj85b\" (UniqueName: \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.772052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.771929 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.871357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.871323 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.872607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.872582 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj85b\" (UniqueName: \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.872695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.872634 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.874914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.874894 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.878298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.878280 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:58.893476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.893448 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj85b\" (UniqueName: \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\") pod \"authorino-6889cbd8c4-p4rvx\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:58.973362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.973289 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\" (UniqueName: \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\") pod \"2742690d-35ea-4b48-8631-cff4b7ab7cd3\" (UID: \"2742690d-35ea-4b48-8631-cff4b7ab7cd3\") " Oct 30 17:21:58.975406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:58.975373 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g" pod "2742690d-35ea-4b48-8631-cff4b7ab7cd3" (UID: "2742690d-35ea-4b48-8631-cff4b7ab7cd3"). InnerVolumeSpecName "kube-api-access-ptx4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:59.025946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.025921 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:21:59.074396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.074365 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptx4g\" (UniqueName: \"kubernetes.io/projected/2742690d-35ea-4b48-8631-cff4b7ab7cd3-kube-api-access-ptx4g\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:21:59.317836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.317804 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:21:59.321060 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:21:59.321033 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a47289_16ea_4d75_b35f_e874e075a353.slice/crio-9af77b3fb018cc68c16447ec9dbafdccf08627ad3b01870e8abb93c74b88baee WatchSource:0}: Error finding container 9af77b3fb018cc68c16447ec9dbafdccf08627ad3b01870e8abb93c74b88baee: Status 404 returned error can't find the container with id 9af77b3fb018cc68c16447ec9dbafdccf08627ad3b01870e8abb93c74b88baee Oct 30 17:21:59.878122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.878019 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxgcv" Oct 30 17:21:59.878122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.878020 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" event={"ID":"56a47289-16ea-4d75-b35f-e874e075a353","Type":"ContainerStarted","Data":"9af77b3fb018cc68c16447ec9dbafdccf08627ad3b01870e8abb93c74b88baee"} Oct 30 17:21:59.947000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.946965 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxgcv"] Oct 30 17:21:59.951052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:21:59.951030 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxgcv"] Oct 30 17:22:00.372382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:00.372350 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2742690d-35ea-4b48-8631-cff4b7ab7cd3" path="/var/lib/kubelet/pods/2742690d-35ea-4b48-8631-cff4b7ab7cd3/volumes" Oct 30 17:22:00.884248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:00.884218 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" event={"ID":"56a47289-16ea-4d75-b35f-e874e075a353","Type":"ContainerStarted","Data":"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea"} Oct 30 17:22:00.921730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:00.921676 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" podStartSLOduration=2.318469985 podStartE2EDuration="2.921659247s" totalImagesPullingTime="603.189262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:00.919979707 +0000 UTC m=+548.697451658" watchObservedRunningTime="2026-10-30 17:22:00.921659247 +0000 UTC m=+548.699131202" Oct 30 17:22:00.948341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:00.948307 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:22:00.948575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:00.948526 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" podUID="325ac348-930d-4a62-9814-0ae65ea6940d" containerName="authorino" containerID="cri-o://b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2" gracePeriod=30 Oct 30 17:22:01.206701 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.206677 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:22:01.293520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.293488 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\" (UniqueName: \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\") pod \"325ac348-930d-4a62-9814-0ae65ea6940d\" (UID: \"325ac348-930d-4a62-9814-0ae65ea6940d\") " Oct 30 17:22:01.295477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.295446 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr" pod "325ac348-930d-4a62-9814-0ae65ea6940d" (UID: "325ac348-930d-4a62-9814-0ae65ea6940d"). InnerVolumeSpecName "kube-api-access-tnsnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:01.394060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.394024 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsnr\" (UniqueName: \"kubernetes.io/projected/325ac348-930d-4a62-9814-0ae65ea6940d-kube-api-access-tnsnr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:22:01.890171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.890137 4622 generic.go:356] "Generic (PLEG): container finished" podID="325ac348-930d-4a62-9814-0ae65ea6940d" containerID="b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2" exitCode=0 Oct 30 17:22:01.890677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.890180 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" event={"ID":"325ac348-930d-4a62-9814-0ae65ea6940d","Type":"ContainerDied","Data":"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2"} Oct 30 17:22:01.890677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.890225 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" event={"ID":"325ac348-930d-4a62-9814-0ae65ea6940d","Type":"ContainerDied","Data":"c5c80f8cc2e59faaca6fbc5f356305a5a9a3a73b4c48f304dd4c22e156166e10"} Oct 30 17:22:01.890677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.890242 4622 scope.go:122] "RemoveContainer" containerID="b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2" Oct 30 17:22:01.890677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.890188 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-df2rd" Oct 30 17:22:01.899043 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.899027 4622 scope.go:122] "RemoveContainer" containerID="b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2" Oct 30 17:22:01.899335 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:22:01.899319 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2\": container with ID starting with b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2 not found: ID does not exist" containerID="b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2" Oct 30 17:22:01.899382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.899346 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2"} err="failed to get container status \"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2\": rpc error: code = NotFound desc = could not find container \"b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2\": container with ID starting with b8f2a3a039b1d7d757529fbe493ef9048dd660a1a65abb29100cedab3c57b2b2 not found: ID does not exist" Oct 30 17:22:01.935416 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.935393 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:22:01.939653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:01.939635 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-df2rd"] Oct 30 17:22:02.373812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:02.373777 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325ac348-930d-4a62-9814-0ae65ea6940d" path="/var/lib/kubelet/pods/325ac348-930d-4a62-9814-0ae65ea6940d/volumes" Oct 30 17:22:08.807571 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.807538 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:22:08.807970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.807947 4622 cpu_manager.go:414] "containerMap: removing container" podUID="325ac348-930d-4a62-9814-0ae65ea6940d" containerName="authorino" Oct 30 17:22:08.807970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.807960 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325ac348-930d-4a62-9814-0ae65ea6940d" containerName="authorino" Oct 30 17:22:08.808053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.808019 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="325ac348-930d-4a62-9814-0ae65ea6940d" containerName="authorino" Oct 30 17:22:08.810610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.810592 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:08.829966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.829946 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:22:08.963066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.963031 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:08.963240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:08.963127 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bbl\" (UniqueName: \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.064113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.064021 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bbl\" (UniqueName: \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.064254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.064166 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.066650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.066627 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.085296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.085246 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bbl\" (UniqueName: \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\") pod \"authorino-bd77cbdbf-zsmdn\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.121122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.121083 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:22:09.411864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.411781 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:22:09.415140 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:22:09.415110 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaabfb15_8283_4a88_9251_a0b21cff0ebc.slice/crio-747f186d1a5fe66b3d360cfa933072e58a8932510581afbbbf813c9b31cb7c48 WatchSource:0}: Error finding container 747f186d1a5fe66b3d360cfa933072e58a8932510581afbbbf813c9b31cb7c48: Status 404 returned error can't find the container with id 747f186d1a5fe66b3d360cfa933072e58a8932510581afbbbf813c9b31cb7c48 Oct 30 17:22:09.931576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.931542 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" event={"ID":"daabfb15-8283-4a88-9251-a0b21cff0ebc","Type":"ContainerStarted","Data":"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759"} Oct 30 17:22:09.931576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.931583 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" event={"ID":"daabfb15-8283-4a88-9251-a0b21cff0ebc","Type":"ContainerStarted","Data":"747f186d1a5fe66b3d360cfa933072e58a8932510581afbbbf813c9b31cb7c48"} Oct 30 17:22:09.971395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:09.971333 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" podStartSLOduration=1.652393758 podStartE2EDuration="1.971320377s" totalImagesPullingTime="318.926619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:09.970887389 +0000 UTC m=+557.748359343" watchObservedRunningTime="2026-10-30 17:22:09.971320377 +0000 UTC m=+557.748792331" Oct 30 17:22:10.000897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.000814 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:22:10.001090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.001065 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" podUID="56a47289-16ea-4d75-b35f-e874e075a353" containerName="authorino" containerID="cri-o://1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea" gracePeriod=30 Oct 30 17:22:10.246563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.246539 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:22:10.377750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.377725 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\" (UniqueName: \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\") pod \"56a47289-16ea-4d75-b35f-e874e075a353\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " Oct 30 17:22:10.377909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.377779 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\" (UniqueName: \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\") pod \"56a47289-16ea-4d75-b35f-e874e075a353\" (UID: \"56a47289-16ea-4d75-b35f-e874e075a353\") " Oct 30 17:22:10.379815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.379789 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b" pod "56a47289-16ea-4d75-b35f-e874e075a353" (UID: "56a47289-16ea-4d75-b35f-e874e075a353"). InnerVolumeSpecName "kube-api-access-zj85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:10.390121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.390091 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert" pod "56a47289-16ea-4d75-b35f-e874e075a353" (UID: "56a47289-16ea-4d75-b35f-e874e075a353"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:10.479452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.479414 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj85b\" (UniqueName: \"kubernetes.io/projected/56a47289-16ea-4d75-b35f-e874e075a353-kube-api-access-zj85b\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.479452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.479450 4622 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/56a47289-16ea-4d75-b35f-e874e075a353-tls-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.937507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.937472 4622 generic.go:356] "Generic (PLEG): container finished" podID="56a47289-16ea-4d75-b35f-e874e075a353" containerID="1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea" exitCode=0 Oct 30 17:22:10.937923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.937515 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" Oct 30 17:22:10.937923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.937558 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" event={"ID":"56a47289-16ea-4d75-b35f-e874e075a353","Type":"ContainerDied","Data":"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea"} Oct 30 17:22:10.937923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.937597 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-p4rvx" event={"ID":"56a47289-16ea-4d75-b35f-e874e075a353","Type":"ContainerDied","Data":"9af77b3fb018cc68c16447ec9dbafdccf08627ad3b01870e8abb93c74b88baee"} Oct 30 17:22:10.937923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.937614 4622 scope.go:122] "RemoveContainer" containerID="1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea" Oct 30 17:22:10.946418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.946365 4622 scope.go:122] "RemoveContainer" containerID="1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea" Oct 30 17:22:10.946630 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:22:10.946614 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea\": container with ID starting with 1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea not found: ID does not exist" containerID="1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea" Oct 30 17:22:10.946692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.946639 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea"} err="failed to get container status \"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea\": rpc error: code = NotFound desc = could not find container \"1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea\": container with ID starting with 1b6c2a72fe4613ef323ca60c9295716e097105984fef76be207e8ab8a131e1ea not found: ID does not exist" Oct 30 17:22:10.986858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.986833 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:22:10.996824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:10.996803 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-p4rvx"] Oct 30 17:22:12.372544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:22:12.372512 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a47289-16ea-4d75-b35f-e874e075a353" path="/var/lib/kubelet/pods/56a47289-16ea-4d75-b35f-e874e075a353/volumes" Oct 30 17:22:24.533107 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:22:24.533031 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:22:54.532742 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:22:54.532702 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:23:13.683419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:13.683376 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:23:13.683872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:13.683454 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:23:16.163670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.163635 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s"] Oct 30 17:23:16.164130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.163993 4622 cpu_manager.go:414] "containerMap: removing container" podUID="56a47289-16ea-4d75-b35f-e874e075a353" containerName="authorino" Oct 30 17:23:16.164130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.164005 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a47289-16ea-4d75-b35f-e874e075a353" containerName="authorino" Oct 30 17:23:16.164130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.164101 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a47289-16ea-4d75-b35f-e874e075a353" containerName="authorino" Oct 30 17:23:16.170669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.167585 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.185732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.185705 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s"] Oct 30 17:23:16.254539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.254503 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c6457c-298e-4c2c-9f30-f6b36738a507-cert\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.254713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.254571 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zmm\" (UniqueName: \"kubernetes.io/projected/55c6457c-298e-4c2c-9f30-f6b36738a507-kube-api-access-x5zmm\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.260513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.260486 4622 generic.go:356] "Generic (PLEG): container finished" podID="586b4908-f577-43ab-9d81-5c163ca66412" containerID="4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c" exitCode=0 Oct 30 17:23:16.260649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.260560 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerDied","Data":"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c"} Oct 30 17:23:16.260885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.260872 4622 scope.go:122] "RemoveContainer" containerID="4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c" Oct 30 17:23:16.261889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.261863 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:23:16.261944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.261927 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:23:16.261983 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.261958 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:23:16.355878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.355825 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c6457c-298e-4c2c-9f30-f6b36738a507-cert\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.356050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.355939 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zmm\" (UniqueName: \"kubernetes.io/projected/55c6457c-298e-4c2c-9f30-f6b36738a507-kube-api-access-x5zmm\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.358411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.358383 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55c6457c-298e-4c2c-9f30-f6b36738a507-cert\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.377545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.377513 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zmm\" (UniqueName: \"kubernetes.io/projected/55c6457c-298e-4c2c-9f30-f6b36738a507-kube-api-access-x5zmm\") pod \"llmisvc-controller-manager-6466c554d8-m9k4s\" (UID: \"55c6457c-298e-4c2c-9f30-f6b36738a507\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.481116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.481036 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:16.785622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.785595 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s"] Oct 30 17:23:16.788539 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:23:16.788513 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c6457c_298e_4c2c_9f30_f6b36738a507.slice/crio-883f4c48dfe67e8d131e04c59b66ecaf0913aeed7cc2df83e1c1349c7c98d4e3 WatchSource:0}: Error finding container 883f4c48dfe67e8d131e04c59b66ecaf0913aeed7cc2df83e1c1349c7c98d4e3: Status 404 returned error can't find the container with id 883f4c48dfe67e8d131e04c59b66ecaf0913aeed7cc2df83e1c1349c7c98d4e3 Oct 30 17:23:16.789940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:16.789923 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:23:17.271496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:17.271460 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" event={"ID":"55c6457c-298e-4c2c-9f30-f6b36738a507","Type":"ContainerStarted","Data":"da26fa53235a5b6a6f74336f069bf1d201999cc95a9473bcf894e14393ccdb97"} Oct 30 17:23:17.271873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:17.271512 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" event={"ID":"55c6457c-298e-4c2c-9f30-f6b36738a507","Type":"ContainerStarted","Data":"883f4c48dfe67e8d131e04c59b66ecaf0913aeed7cc2df83e1c1349c7c98d4e3"} Oct 30 17:23:17.271873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:17.271645 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:17.273553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:17.273535 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerStarted","Data":"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19"} Oct 30 17:23:17.312761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:17.312702 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" podStartSLOduration=0.927652733 podStartE2EDuration="1.31268506s" totalImagesPullingTime="385.032327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:17.311283106 +0000 UTC m=+625.088755063" watchObservedRunningTime="2026-10-30 17:23:17.31268506 +0000 UTC m=+625.090157019" Oct 30 17:23:18.139209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:18.139171 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:23:18.139539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:18.139509 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 17:23:18.139659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:18.139558 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 17:23:19.139069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:19.139039 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 17:23:19.139494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:19.139089 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 17:23:20.139130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:20.139093 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 17:23:20.139551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:20.139152 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 17:23:21.139373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:21.139337 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 17:23:21.139866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:21.139410 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 17:23:22.138894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:22.138857 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 17:23:22.139073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:22.138915 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 17:23:23.143465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:23.143425 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:23:23.143988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:23.143967 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:23:23.144554 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:23.144532 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:23:24.533188 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:23:24.533152 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:23:48.281164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.281133 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-m9k4s" Oct 30 17:23:48.353783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.353741 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:23:48.354044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.353996 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" podUID="836205fb-512c-432c-9c80-cafba4f7f732" containerName="manager" containerID="cri-o://38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66" gracePeriod=30 Oct 30 17:23:48.594563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.594537 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:23:48.757615 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.757580 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\" (UniqueName: \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\") pod \"836205fb-512c-432c-9c80-cafba4f7f732\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " Oct 30 17:23:48.757780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.757637 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\" (UniqueName: \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\") pod \"836205fb-512c-432c-9c80-cafba4f7f732\" (UID: \"836205fb-512c-432c-9c80-cafba4f7f732\") " Oct 30 17:23:48.759826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.759796 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769" pod "836205fb-512c-432c-9c80-cafba4f7f732" (UID: "836205fb-512c-432c-9c80-cafba4f7f732"). InnerVolumeSpecName "kube-api-access-86769". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:48.759826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.759807 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert" pod "836205fb-512c-432c-9c80-cafba4f7f732" (UID: "836205fb-512c-432c-9c80-cafba4f7f732"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:48.858622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.858529 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86769\" (UniqueName: \"kubernetes.io/projected/836205fb-512c-432c-9c80-cafba4f7f732-kube-api-access-86769\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:23:48.858622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:48.858568 4622 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/836205fb-512c-432c-9c80-cafba4f7f732-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:23:49.436429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.436392 4622 generic.go:356] "Generic (PLEG): container finished" podID="836205fb-512c-432c-9c80-cafba4f7f732" containerID="38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66" exitCode=0 Oct 30 17:23:49.436911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.436455 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" Oct 30 17:23:49.436911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.436484 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" event={"ID":"836205fb-512c-432c-9c80-cafba4f7f732","Type":"ContainerDied","Data":"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66"} Oct 30 17:23:49.436911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.436547 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r" event={"ID":"836205fb-512c-432c-9c80-cafba4f7f732","Type":"ContainerDied","Data":"615f39e6686508b1c1f0e86a0411296f5cf537ab2a0a872ec46a201bf7f74db0"} Oct 30 17:23:49.436911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.436579 4622 scope.go:122] "RemoveContainer" containerID="38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66" Oct 30 17:23:49.446130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.446115 4622 scope.go:122] "RemoveContainer" containerID="38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66" Oct 30 17:23:49.446428 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:23:49.446408 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66\": container with ID starting with 38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66 not found: ID does not exist" containerID="38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66" Oct 30 17:23:49.446493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.446436 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66"} err="failed to get container status \"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66\": rpc error: code = NotFound desc = could not find container \"38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66\": container with ID starting with 38c586531ce9583cea83f9ab007b9b471da8c314ff3a05d02c8d2cc56a14dc66 not found: ID does not exist" Oct 30 17:23:49.488251 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.488223 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:23:49.500550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:49.500523 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-js49r"] Oct 30 17:23:50.373240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:23:50.373206 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836205fb-512c-432c-9c80-cafba4f7f732" path="/var/lib/kubelet/pods/836205fb-512c-432c-9c80-cafba4f7f732/volumes" Oct 30 17:23:54.533397 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:23:54.533360 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:24:24.183096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:24.183054 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:24:24.183718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:24.183120 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:24:24.532769 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:24:24.532725 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:24:37.137824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.137788 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:24:37.138371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.138056 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" podUID="daabfb15-8283-4a88-9251-a0b21cff0ebc" containerName="authorino" containerID="cri-o://d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759" gracePeriod=30 Oct 30 17:24:37.385274 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.385242 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:24:37.501402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.501315 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\" (UniqueName: \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\") pod \"daabfb15-8283-4a88-9251-a0b21cff0ebc\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " Oct 30 17:24:37.501402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.501354 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\" (UniqueName: \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\") pod \"daabfb15-8283-4a88-9251-a0b21cff0ebc\" (UID: \"daabfb15-8283-4a88-9251-a0b21cff0ebc\") " Oct 30 17:24:37.503482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.503452 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl" pod "daabfb15-8283-4a88-9251-a0b21cff0ebc" (UID: "daabfb15-8283-4a88-9251-a0b21cff0ebc"). InnerVolumeSpecName "kube-api-access-d4bbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:37.511209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.511181 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert" pod "daabfb15-8283-4a88-9251-a0b21cff0ebc" (UID: "daabfb15-8283-4a88-9251-a0b21cff0ebc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:37.602637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.602602 4622 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/daabfb15-8283-4a88-9251-a0b21cff0ebc-tls-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:24:37.602637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.602632 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4bbl\" (UniqueName: \"kubernetes.io/projected/daabfb15-8283-4a88-9251-a0b21cff0ebc-kube-api-access-d4bbl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:24:37.709093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.709064 4622 generic.go:356] "Generic (PLEG): container finished" podID="daabfb15-8283-4a88-9251-a0b21cff0ebc" containerID="d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759" exitCode=0 Oct 30 17:24:37.709277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.709109 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" Oct 30 17:24:37.709277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.709142 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" event={"ID":"daabfb15-8283-4a88-9251-a0b21cff0ebc","Type":"ContainerDied","Data":"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759"} Oct 30 17:24:37.709277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.709182 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd77cbdbf-zsmdn" event={"ID":"daabfb15-8283-4a88-9251-a0b21cff0ebc","Type":"ContainerDied","Data":"747f186d1a5fe66b3d360cfa933072e58a8932510581afbbbf813c9b31cb7c48"} Oct 30 17:24:37.709277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.709198 4622 scope.go:122] "RemoveContainer" containerID="d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759" Oct 30 17:24:37.718171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.718154 4622 scope.go:122] "RemoveContainer" containerID="d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759" Oct 30 17:24:37.718435 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:24:37.718417 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759\": container with ID starting with d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759 not found: ID does not exist" containerID="d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759" Oct 30 17:24:37.718496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.718447 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759"} err="failed to get container status \"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759\": rpc error: code = NotFound desc = could not find container \"d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759\": container with ID starting with d1b0228a26d968959afafd9cd4a32374ceb55128e171b55a240ef26389909759 not found: ID does not exist" Oct 30 17:24:37.760372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.760314 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:24:37.766376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:37.766355 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bd77cbdbf-zsmdn"] Oct 30 17:24:38.373106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:38.373074 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daabfb15-8283-4a88-9251-a0b21cff0ebc" path="/var/lib/kubelet/pods/daabfb15-8283-4a88-9251-a0b21cff0ebc/volumes" Oct 30 17:24:41.847540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847510 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:24:41.847929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847882 4622 cpu_manager.go:414] "containerMap: removing container" podUID="daabfb15-8283-4a88-9251-a0b21cff0ebc" containerName="authorino" Oct 30 17:24:41.847929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847892 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daabfb15-8283-4a88-9251-a0b21cff0ebc" containerName="authorino" Oct 30 17:24:41.847929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847915 4622 cpu_manager.go:414] "containerMap: removing container" podUID="836205fb-512c-432c-9c80-cafba4f7f732" containerName="manager" Oct 30 17:24:41.847929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847920 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836205fb-512c-432c-9c80-cafba4f7f732" containerName="manager" Oct 30 17:24:41.848059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.847990 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="daabfb15-8283-4a88-9251-a0b21cff0ebc" containerName="authorino" Oct 30 17:24:41.848059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.848000 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="836205fb-512c-432c-9c80-cafba4f7f732" containerName="manager" Oct 30 17:24:41.852186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.852171 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.880827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.880796 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:24:41.942984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.942954 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943009 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943065 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943104 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943125 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943181 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943244 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943301 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72skj\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:41.943425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:41.943379 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044532 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044569 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044597 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044627 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72skj\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044652 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044682 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044751 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.044791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044772 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.044798 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.045078 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.045159 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.045331 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.045377 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.045418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.045390 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.047108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.047086 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.047345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.047327 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.080410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.080387 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.080564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.080475 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72skj\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\") pod \"maas-default-gateway-openshift-default-7df854bf67-fp8cd\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.166108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.166025 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:42.478125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.478098 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:24:42.479722 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:24:42.479698 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aef45da_4bf0_42b1_a5de_800f71aa1711.slice/crio-1f7a65b030188bbac078b45941f2dfc5768e73003a71a89b0af496dc7073ed35 WatchSource:0}: Error finding container 1f7a65b030188bbac078b45941f2dfc5768e73003a71a89b0af496dc7073ed35: Status 404 returned error can't find the container with id 1f7a65b030188bbac078b45941f2dfc5768e73003a71a89b0af496dc7073ed35 Oct 30 17:24:42.481678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.481647 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:42.481730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.481721 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:42.481768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.481749 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:24:42.739495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.739407 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" event={"ID":"2aef45da-4bf0-42b1-a5de-800f71aa1711","Type":"ContainerStarted","Data":"0e5f912147a1cdb083c819c455284ccf2839caef18012fb4db0964af24354d21"} Oct 30 17:24:42.739495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.739475 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" event={"ID":"2aef45da-4bf0-42b1-a5de-800f71aa1711","Type":"ContainerStarted","Data":"1f7a65b030188bbac078b45941f2dfc5768e73003a71a89b0af496dc7073ed35"} Oct 30 17:24:42.798834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:42.798782 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podStartSLOduration=1.798766878 podStartE2EDuration="1.798766878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:42.79776596 +0000 UTC m=+710.575237948" watchObservedRunningTime="2026-10-30 17:24:42.798766878 +0000 UTC m=+710.576238833" Oct 30 17:24:43.166549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:43.166510 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:43.168109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:43.168082 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:43.168237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:43.168139 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:44.166726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:44.166687 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:44.167166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:44.166744 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:45.167195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:45.167154 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:45.167670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:45.167217 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:46.166960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:46.166920 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:46.167143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:46.166981 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:47.166761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:47.166727 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:47.167178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:47.166778 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:48.166531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:48.166488 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:48.166714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:48.166553 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:49.167213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:49.167171 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 17:24:49.167643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:49.167237 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 17:24:50.700677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:50.699736 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:50.700677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:50.700557 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:50.701528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:24:50.701508 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:24:54.533208 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:24:54.533171 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:25:24.533247 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:25:24.533162 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:25:54.532611 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:25:54.532573 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:26:24.533363 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:26:24.533326 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:26:27.887087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.887053 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 17:26:27.890524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.890501 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:27.890794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.890769 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czzjj\" (UniqueName: \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:27.890862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.890840 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:27.905415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.905397 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-db98b0\"" Oct 30 17:26:27.917396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.917374 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-db98b0-dockercfg-cchxz\"" Oct 30 17:26:27.919499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.919476 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 17:26:27.992133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.992098 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czzjj\" (UniqueName: \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:27.992133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.992139 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:27.993040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:27.993017 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:28.015987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:28.015963 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czzjj\" (UniqueName: \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\") pod \"payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:28.203368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:28.203282 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:28.503195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:28.503172 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 17:26:28.504524 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:28.504494 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d5375_3e23_44a4_83a3_b97150984c71.slice/crio-0d04d5fb8f1c8fcb9b14743ee030e2c7a13f7851175a1c4507dcc03fb98674da WatchSource:0}: Error finding container 0d04d5fb8f1c8fcb9b14743ee030e2c7a13f7851175a1c4507dcc03fb98674da: Status 404 returned error can't find the container with id 0d04d5fb8f1c8fcb9b14743ee030e2c7a13f7851175a1c4507dcc03fb98674da Oct 30 17:26:29.302907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:29.302866 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" event={"ID":"998d5375-3e23-44a4-83a3-b97150984c71","Type":"ContainerStarted","Data":"0d04d5fb8f1c8fcb9b14743ee030e2c7a13f7851175a1c4507dcc03fb98674da"} Oct 30 17:26:31.117179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.117143 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 17:26:31.120723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.120707 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.139553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.139527 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 17:26:31.149701 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.149680 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-306aed\"" Oct 30 17:26:31.149822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.149707 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-306aed-dockercfg-sgvlk\"" Oct 30 17:26:31.221131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.221068 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.221331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.221213 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.315395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.315364 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" event={"ID":"998d5375-3e23-44a4-83a3-b97150984c71","Type":"ContainerStarted","Data":"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4"} Oct 30 17:26:31.315587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.315499 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:31.322010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.321986 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.322145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.322119 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.322889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.322872 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.347108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.347083 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\") pod \"payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.361047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.361008 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" podStartSLOduration=2.446322591 podStartE2EDuration="4.360997077s" totalImagesPullingTime="1.914674486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.359171696 +0000 UTC m=+819.136643652" watchObservedRunningTime="2026-10-30 17:26:31.360997077 +0000 UTC m=+819.138469031" Oct 30 17:26:31.433130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.433066 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:31.730060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.730037 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 17:26:31.731884 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:31.731853 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b362a8b_9719_4a90_ba33_81417c635064.slice/crio-a1c412dbf7a7db18fbd4de1b156c2dbf82bbd3c8b355407ca0c0bc807672060e WatchSource:0}: Error finding container a1c412dbf7a7db18fbd4de1b156c2dbf82bbd3c8b355407ca0c0bc807672060e: Status 404 returned error can't find the container with id a1c412dbf7a7db18fbd4de1b156c2dbf82bbd3c8b355407ca0c0bc807672060e Oct 30 17:26:31.962495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.962450 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:26:31.966671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.966647 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:31.987458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:31.987398 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8d946d-openshift-default-dockercfg-txjzw\"" Oct 30 17:26:32.001719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.001693 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:26:32.027830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.027795 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.027830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.027830 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.027845 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028004 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028033 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028051 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qx74\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028108 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028177 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.028349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.028195 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.128858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128822 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128874 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128898 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qx74\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128938 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128972 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.128998 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129200 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129247 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129302 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129349 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129379 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129450 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129543 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.129892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.129845 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.131350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.131328 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.131454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.131443 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.152690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.152644 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.152859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.152781 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qx74\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\") pod \"e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.280480 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.280397 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:32.322931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.322896 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" event={"ID":"3b362a8b-9719-4a90-ba33-81417c635064","Type":"ContainerStarted","Data":"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805"} Oct 30 17:26:32.323086 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.322944 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" event={"ID":"3b362a8b-9719-4a90-ba33-81417c635064","Type":"ContainerStarted","Data":"a1c412dbf7a7db18fbd4de1b156c2dbf82bbd3c8b355407ca0c0bc807672060e"} Oct 30 17:26:32.365652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.365603 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" podStartSLOduration=1.365589001 podStartE2EDuration="1.365589001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:32.364820784 +0000 UTC m=+820.142292750" watchObservedRunningTime="2026-10-30 17:26:32.365589001 +0000 UTC m=+820.143060955" Oct 30 17:26:32.597493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.597465 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:26:32.598122 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:32.598097 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8296a44a_4f3f_4155_84c2_ac7ff45044d8.slice/crio-4a0621833f7a517adf94f3c8a5aafc7b7bd6740f7e95967323d32412be3aafa5 WatchSource:0}: Error finding container 4a0621833f7a517adf94f3c8a5aafc7b7bd6740f7e95967323d32412be3aafa5: Status 404 returned error can't find the container with id 4a0621833f7a517adf94f3c8a5aafc7b7bd6740f7e95967323d32412be3aafa5 Oct 30 17:26:32.604077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.604045 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:26:32.604179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.604112 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:26:32.604179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:32.604148 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:26:33.330212 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:33.330174 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerStarted","Data":"e727db0e0bea54e53e7b89f0f6ab41a5984812a6a631021e7f0f6a111fd9b43d"} Oct 30 17:26:33.330659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:33.330256 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerStarted","Data":"4a0621833f7a517adf94f3c8a5aafc7b7bd6740f7e95967323d32412be3aafa5"} Oct 30 17:26:33.330659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:33.330491 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:33.378490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:33.378440 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podStartSLOduration=2.378427932 podStartE2EDuration="2.378427932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:33.376498172 +0000 UTC m=+821.153970126" watchObservedRunningTime="2026-10-30 17:26:33.378427932 +0000 UTC m=+821.155899886" Oct 30 17:26:34.280704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:34.280673 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:34.285491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:34.285467 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:34.336506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:34.336473 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:34.337309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:34.337289 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:26:36.882761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.882728 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 17:26:36.886351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.886335 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:36.919409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.919378 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-306aed-serving-cert\"" Oct 30 17:26:36.919619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.919453 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:26:36.919619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.919472 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:36.919619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.919503 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-s8xph\"" Oct 30 17:26:36.919619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.919585 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:36.921758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.921732 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 17:26:36.975913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.975875 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:36.975913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.975918 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:36.976114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.975952 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpww\" (UniqueName: \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:36.976114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:36.976044 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.076629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.076591 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.076821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.076639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.076821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.076686 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpww\" (UniqueName: \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.076821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.076716 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.077406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.077383 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.079145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.079124 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.079246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.079195 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.100967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.100943 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpww\" (UniqueName: \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\") pod \"maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.203925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.203850 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:37.503086 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:37.503051 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 17:26:37.504200 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:37.504174 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65c9f56_db3d_4ab3_85d4_25f1c4f72684.slice/crio-2a491d0b2e645f13c93e212735b2e86e705c19d17bf7f90aa2d33abde7d27262 WatchSource:0}: Error finding container 2a491d0b2e645f13c93e212735b2e86e705c19d17bf7f90aa2d33abde7d27262: Status 404 returned error can't find the container with id 2a491d0b2e645f13c93e212735b2e86e705c19d17bf7f90aa2d33abde7d27262 Oct 30 17:26:38.374706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:38.374665 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" event={"ID":"a65c9f56-db3d-4ab3-85d4-25f1c4f72684","Type":"ContainerStarted","Data":"2a491d0b2e645f13c93e212735b2e86e705c19d17bf7f90aa2d33abde7d27262"} Oct 30 17:26:40.378576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:40.378541 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" event={"ID":"a65c9f56-db3d-4ab3-85d4-25f1c4f72684","Type":"ContainerStarted","Data":"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd"} Oct 30 17:26:40.378982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:40.378601 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:40.423150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:40.423091 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" podStartSLOduration=2.088241958 podStartE2EDuration="4.423075747s" totalImagesPullingTime="2.334833789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:40.421397202 +0000 UTC m=+828.198869157" watchObservedRunningTime="2026-10-30 17:26:40.423075747 +0000 UTC m=+828.200547701" Oct 30 17:26:42.326334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:42.326303 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 17:26:44.338832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:44.338802 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 17:26:46.389704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:46.389678 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 17:26:50.686616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.686582 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 17:26:50.691312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.691290 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.718942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.718913 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0e7403\"" Oct 30 17:26:50.719078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.719001 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0e7403-dockercfg-v6pdx\"" Oct 30 17:26:50.722557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.722532 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 17:26:50.784035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.784002 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 17:26:50.788306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.788284 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:50.804582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.804557 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.804714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.804633 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkl9\" (UniqueName: \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.809617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.809595 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 17:26:50.905607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.905535 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkl9\" (UniqueName: \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.905607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.905598 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:50.905804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.905632 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.905804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.905660 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxwm\" (UniqueName: \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:50.906697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.906678 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:50.929063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:50.929033 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkl9\" (UniqueName: \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\") pod \"payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:51.006726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.006648 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:51.006726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.006688 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxwm\" (UniqueName: \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:51.007377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.007360 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:51.030756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.030726 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:51.031073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.031053 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxwm\" (UniqueName: \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\") pod \"payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:51.101628 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.101599 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:51.325376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.325348 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 17:26:51.326234 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:51.326207 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8551c6_73d7_40a8_ba0e_4722339d7a9c.slice/crio-6619be461765c0f1a63e709a095d64f44231038e1afa318394c61aac92de8e88 WatchSource:0}: Error finding container 6619be461765c0f1a63e709a095d64f44231038e1afa318394c61aac92de8e88: Status 404 returned error can't find the container with id 6619be461765c0f1a63e709a095d64f44231038e1afa318394c61aac92de8e88 Oct 30 17:26:51.401544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.401520 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 17:26:51.402778 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:26:51.402753 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564b66d8_b77c_4fb2_b34a_69e15936452e.slice/crio-e2d0f3b72882be01044eeb77675481abef02c77c62ec3ae3cc1a5f5e1448e965 WatchSource:0}: Error finding container e2d0f3b72882be01044eeb77675481abef02c77c62ec3ae3cc1a5f5e1448e965: Status 404 returned error can't find the container with id e2d0f3b72882be01044eeb77675481abef02c77c62ec3ae3cc1a5f5e1448e965 Oct 30 17:26:51.439501 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.439469 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" event={"ID":"564b66d8-b77c-4fb2-b34a-69e15936452e","Type":"ContainerStarted","Data":"e2d0f3b72882be01044eeb77675481abef02c77c62ec3ae3cc1a5f5e1448e965"} Oct 30 17:26:51.441469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.441438 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" event={"ID":"dc8551c6-73d7-40a8-ba0e-4722339d7a9c","Type":"ContainerStarted","Data":"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0"} Oct 30 17:26:51.441617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.441496 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" event={"ID":"dc8551c6-73d7-40a8-ba0e-4722339d7a9c","Type":"ContainerStarted","Data":"6619be461765c0f1a63e709a095d64f44231038e1afa318394c61aac92de8e88"} Oct 30 17:26:51.442320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.442301 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:26:51.489436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:51.489386 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" podStartSLOduration=1.489370392 podStartE2EDuration="1.489370392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:51.487420277 +0000 UTC m=+839.264892247" watchObservedRunningTime="2026-10-30 17:26:51.489370392 +0000 UTC m=+839.266842347" Oct 30 17:26:52.449225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:52.449171 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" event={"ID":"564b66d8-b77c-4fb2-b34a-69e15936452e","Type":"ContainerStarted","Data":"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18"} Oct 30 17:26:52.494083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:52.493996 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" podStartSLOduration=2.493975984 podStartE2EDuration="2.493975984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:52.491681353 +0000 UTC m=+840.269153308" watchObservedRunningTime="2026-10-30 17:26:52.493975984 +0000 UTC m=+840.271447939" Oct 30 17:26:53.455818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:26:53.455789 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:26:54.533535 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:26:54.533497 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:27:01.489427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.489388 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 17:27:01.495919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.495897 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.512926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.512900 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 17:27:01.515386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.515360 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0e7403-serving-cert\"" Oct 30 17:27:01.637001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.636961 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.637001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.637006 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.637243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.637129 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.637243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.637184 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57tqg\" (UniqueName: \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.739107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.738847 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.739107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.738915 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57tqg\" (UniqueName: \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.739107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.738994 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.739107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.739024 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.740826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.740758 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.741991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.741965 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.742733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.742709 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.762896 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.762863 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57tqg\" (UniqueName: \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\") pod \"maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:01.817423 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:01.817396 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:02.131797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.131770 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 17:27:02.132769 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:27:02.132742 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2748d16_0769_42cd_8ac8_3999a15c6b19.slice/crio-09d5c4d3efd1dd239529175c4f095feb01079634514b840d67d9a0ee9e315003 WatchSource:0}: Error finding container 09d5c4d3efd1dd239529175c4f095feb01079634514b840d67d9a0ee9e315003: Status 404 returned error can't find the container with id 09d5c4d3efd1dd239529175c4f095feb01079634514b840d67d9a0ee9e315003 Oct 30 17:27:02.518476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.518392 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" event={"ID":"a2748d16-0769-42cd-8ac8-3999a15c6b19","Type":"ContainerStarted","Data":"09d5c4d3efd1dd239529175c4f095feb01079634514b840d67d9a0ee9e315003"} Oct 30 17:27:02.528403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.528373 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 17:27:02.532378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.532353 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.563691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.560327 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"default-dockercfg-nc6f8\"" Oct 30 17:27:02.563691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.560327 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"simulator-w0-27567a-kserve-self-signed-certs\"" Oct 30 17:27:02.563691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.563300 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 17:27:02.577881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.577863 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"openshift-service-ca.crt\"" Oct 30 17:27:02.577982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.577903 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"kube-root-ca.crt\"" Oct 30 17:27:02.649365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649336 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.649507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649471 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.649575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649532 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.649575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649563 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzx8\" (UniqueName: \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.649777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649629 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.649777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.649654 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751229 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751325 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751367 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751387 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzx8\" (UniqueName: \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751438 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751457 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.751732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.751710 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.752243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.752213 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.752407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.752249 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.753872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.753849 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.753991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.753975 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.774996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.774938 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzx8\" (UniqueName: \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\") pod \"simulator-w0-27567a-kserve-5747f855d7-5twnj\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:02.850402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:02.850258 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:03.156509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.156472 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 17:27:03.158481 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:27:03.158450 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd294de07_d345_411b_ae06_c23263f5c261.slice/crio-fa5c04920bdb8a90dfdfcd7ed0497a0d238fc4533a2d0dc4f2d5070e8f0b1613 WatchSource:0}: Error finding container fa5c04920bdb8a90dfdfcd7ed0497a0d238fc4533a2d0dc4f2d5070e8f0b1613: Status 404 returned error can't find the container with id fa5c04920bdb8a90dfdfcd7ed0497a0d238fc4533a2d0dc4f2d5070e8f0b1613 Oct 30 17:27:03.459578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.459498 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 17:27:03.536963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.535709 4622 generic.go:356] "Generic (PLEG): container finished" podID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerID="e727db0e0bea54e53e7b89f0f6ab41a5984812a6a631021e7f0f6a111fd9b43d" exitCode=0 Oct 30 17:27:03.536963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.535795 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerDied","Data":"e727db0e0bea54e53e7b89f0f6ab41a5984812a6a631021e7f0f6a111fd9b43d"} Oct 30 17:27:03.536963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.536220 4622 scope.go:122] "RemoveContainer" containerID="e727db0e0bea54e53e7b89f0f6ab41a5984812a6a631021e7f0f6a111fd9b43d" Oct 30 17:27:03.539305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.538510 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:27:03.539305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.538645 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:27:03.539305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.538693 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:27:03.543101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.543072 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" event={"ID":"a2748d16-0769-42cd-8ac8-3999a15c6b19","Type":"ContainerStarted","Data":"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9"} Oct 30 17:27:03.543656 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.543634 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:03.549550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.545459 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" event={"ID":"d294de07-d345-411b-ae06-c23263f5c261","Type":"ContainerStarted","Data":"fa5c04920bdb8a90dfdfcd7ed0497a0d238fc4533a2d0dc4f2d5070e8f0b1613"} Oct 30 17:27:03.627464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:03.626192 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" podStartSLOduration=2.200088857 podStartE2EDuration="2.62617588s" totalImagesPullingTime="426.087023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:03.624931348 +0000 UTC m=+851.402403308" watchObservedRunningTime="2026-10-30 17:27:03.62617588 +0000 UTC m=+851.403647839" Oct 30 17:27:04.337278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:04.337240 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:27:04.480034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:04.479999 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 17:27:04.566226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:04.566189 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerStarted","Data":"e978c0bee3baa09de390681632c4e8657bdfcbb8b3c1933a989f24ef7a29421e"} Oct 30 17:27:05.281256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.281225 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:27:05.282491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.282468 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 17:27:05.282617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.282522 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 17:27:05.576645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.576606 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" event={"ID":"d294de07-d345-411b-ae06-c23263f5c261","Type":"ContainerStarted","Data":"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04"} Oct 30 17:27:05.621429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.621369 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podStartSLOduration=1.5488080659999999 podStartE2EDuration="3.6213542s" totalImagesPullingTime="2.072546134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:05.620018437 +0000 UTC m=+853.397490393" watchObservedRunningTime="2026-10-30 17:27:05.6213542 +0000 UTC m=+853.398826154" Oct 30 17:27:05.852343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:05.850380 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:06.281037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:06.280960 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 17:27:06.281037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:06.281011 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 17:27:07.281224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:07.281194 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 17:27:07.281713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:07.281246 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 17:27:08.281415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:08.281381 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 17:27:08.281828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:08.281435 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 17:27:09.280792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:09.280760 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 17:27:09.280988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:09.280821 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 17:27:10.287864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:10.287834 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:27:10.288393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:10.288287 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:27:10.291091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:10.291069 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:27:10.582434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:10.582358 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 17:27:12.851073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:12.851041 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:12.864476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:12.864453 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:13.628983 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:13.628957 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:27:24.532597 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:27:24.532554 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:27:28.805430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.805393 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 17:27:28.808979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.808957 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.824634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.824615 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"default-dockercfg-89cmx\"" Oct 30 17:27:28.837145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.837114 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"kube-root-ca.crt\"" Oct 30 17:27:28.837145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.837119 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"openshift-service-ca.crt\"" Oct 30 17:27:28.839723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.839700 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 17:27:28.915402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.915374 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"simulator-w2-506124-kserve-self-signed-certs\"" Oct 30 17:27:28.928840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.928805 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.928840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.928840 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.929073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.928953 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8kj\" (UniqueName: \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.929073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.929020 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.929073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.929049 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:28.929073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:28.929072 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030541 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030582 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030619 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8kj\" (UniqueName: \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030646 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030674 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.030797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.030699 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.031190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.031160 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.031315 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.031228 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.031385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.031317 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.032920 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.032894 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.033154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.033140 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.056446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.055505 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8kj\" (UniqueName: \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\") pod \"simulator-w2-506124-kserve-5ffc84887f-whpsf\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.121650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.121615 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:29.420229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.420203 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 17:27:29.422360 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:27:29.422333 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559092cc_213f_45b7_aeb6_6aa816959162.slice/crio-2eda88d3a1f81d91ed33c5202d89ef233d544b50c461e4a3fd5aa07ba96f6455 WatchSource:0}: Error finding container 2eda88d3a1f81d91ed33c5202d89ef233d544b50c461e4a3fd5aa07ba96f6455: Status 404 returned error can't find the container with id 2eda88d3a1f81d91ed33c5202d89ef233d544b50c461e4a3fd5aa07ba96f6455 Oct 30 17:27:29.719400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.719304 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" event={"ID":"559092cc-213f-45b7-aeb6-6aa816959162","Type":"ContainerStarted","Data":"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef"} Oct 30 17:27:29.719400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.719357 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" event={"ID":"559092cc-213f-45b7-aeb6-6aa816959162","Type":"ContainerStarted","Data":"2eda88d3a1f81d91ed33c5202d89ef233d544b50c461e4a3fd5aa07ba96f6455"} Oct 30 17:27:29.762318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:29.762256 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podStartSLOduration=1.7622439650000001 podStartE2EDuration="1.762243965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:29.760493471 +0000 UTC m=+877.537965425" watchObservedRunningTime="2026-10-30 17:27:29.762243965 +0000 UTC m=+877.539715964" Oct 30 17:27:30.122577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:30.122545 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:39.122206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:39.122173 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:39.134932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:39.134906 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:39.785875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:39.785848 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:27:42.574961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.574928 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 17:27:42.581930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.581905 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.595677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.595643 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 17:27:42.604779 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.604759 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-27567a\"/\"premium-w0-27567a-kserve-self-signed-certs\"" Oct 30 17:27:42.673404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673362 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.673404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673401 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.673654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673426 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.673654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673532 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.673654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673586 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.673819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.673683 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-698zw\" (UniqueName: \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774511 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774556 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774622 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774689 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774750 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-698zw\" (UniqueName: \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.774803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.774778 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.775149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.775035 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.775246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.775162 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.775644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.775619 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.777447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.777424 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.777622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.777516 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.799930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.799889 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-698zw\" (UniqueName: \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\") pod \"premium-w0-27567a-kserve-85f886465d-c7xtf\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:42.901355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:42.901257 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:43.201590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:43.201563 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 17:27:43.202092 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:27:43.202065 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbf0b8f4_42ed_4167_9777_ec1f917d16b6.slice/crio-5278132e8ffedb9a7ee2fb771fc0dfd86a460ed97ba4bd02a0926f3242d21949 WatchSource:0}: Error finding container 5278132e8ffedb9a7ee2fb771fc0dfd86a460ed97ba4bd02a0926f3242d21949: Status 404 returned error can't find the container with id 5278132e8ffedb9a7ee2fb771fc0dfd86a460ed97ba4bd02a0926f3242d21949 Oct 30 17:27:43.798483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:43.798446 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" event={"ID":"fbf0b8f4-42ed-4167-9777-ec1f917d16b6","Type":"ContainerStarted","Data":"62a611d1503b212ee7798b9674a70e7d6735a373028f4a5ff16ab4cc245ce7da"} Oct 30 17:27:43.798878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:43.798497 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" event={"ID":"fbf0b8f4-42ed-4167-9777-ec1f917d16b6","Type":"ContainerStarted","Data":"5278132e8ffedb9a7ee2fb771fc0dfd86a460ed97ba4bd02a0926f3242d21949"} Oct 30 17:27:43.841926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:43.841873 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podStartSLOduration=1.841859013 podStartE2EDuration="1.841859013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:43.839522921 +0000 UTC m=+891.616994877" watchObservedRunningTime="2026-10-30 17:27:43.841859013 +0000 UTC m=+891.619330968" Oct 30 17:27:43.902141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:43.902099 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:52.880970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.880932 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 17:27:52.885583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.885561 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.901636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.901612 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:52.913551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.913527 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"kube-root-ca.crt\"" Oct 30 17:27:52.913687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.913605 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"openshift-service-ca.crt\"" Oct 30 17:27:52.913687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.913633 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"default-dockercfg-4sfjv\"" Oct 30 17:27:52.913778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.913688 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"premium-w3-306aed-kserve-self-signed-certs\"" Oct 30 17:27:52.914734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.914713 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:52.916162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.916144 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 17:27:52.973377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973348 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9mw\" (UniqueName: \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.973564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973415 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.973564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973506 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.973680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973595 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.973741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973718 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:52.973797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:52.973750 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.074732 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.074782 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.074864 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9mw\" (UniqueName: \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.074949 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.074979 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.075011 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.075491 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.076063 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.077542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.076105 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.079516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.079489 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.093923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.093828 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.099601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.099575 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9mw\" (UniqueName: \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\") pod \"premium-w3-306aed-kserve-855989f9c8-652rb\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.198527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.198451 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:53.494740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.494711 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 17:27:53.495762 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:27:53.495735 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8c80672_f796_4050_bcfe_c274eb4c805d.slice/crio-08bdf243c68245e6d60869022597a709af7047b614f818178f801eb62565b67d WatchSource:0}: Error finding container 08bdf243c68245e6d60869022597a709af7047b614f818178f801eb62565b67d: Status 404 returned error can't find the container with id 08bdf243c68245e6d60869022597a709af7047b614f818178f801eb62565b67d Oct 30 17:27:53.852981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.852946 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" event={"ID":"f8c80672-f796-4050-bcfe-c274eb4c805d","Type":"ContainerStarted","Data":"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559"} Oct 30 17:27:53.853175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.852990 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" event={"ID":"f8c80672-f796-4050-bcfe-c274eb4c805d","Type":"ContainerStarted","Data":"08bdf243c68245e6d60869022597a709af7047b614f818178f801eb62565b67d"} Oct 30 17:27:53.864455 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.864434 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:27:53.894927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:53.894884 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podStartSLOduration=1.8948722519999999 podStartE2EDuration="1.894872252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:53.893933869 +0000 UTC m=+901.671405824" watchObservedRunningTime="2026-10-30 17:27:53.894872252 +0000 UTC m=+901.672344206" Oct 30 17:27:54.198945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:27:54.198870 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:27:54.533252 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:27:54.533212 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:28:03.198594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:03.198547 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:28:03.210566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:03.210540 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:28:03.923157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:03.923123 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:28:21.646828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.646745 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 17:28:21.650984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.650964 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669595 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wcw\" (UniqueName: \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669635 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669686 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669705 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669740 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.669767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.669757 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.673193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.673169 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 17:28:21.681491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.681467 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"openshift-service-ca.crt\"" Oct 30 17:28:21.681608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.681554 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"default-dockercfg-jgc4t\"" Oct 30 17:28:21.769566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.769521 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"kube-root-ca.crt\"" Oct 30 17:28:21.769740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.769576 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"premium-w4-0e7403-kserve-self-signed-certs\"" Oct 30 17:28:21.771095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771078 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771112 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771207 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771192 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771224 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771319 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wcw\" (UniqueName: \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771365 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771648 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771708 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.771865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.771769 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.773786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.773758 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.773904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.773818 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.794610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.794575 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wcw\" (UniqueName: \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\") pod \"premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:21.966208 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:21.966111 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:22.263734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:22.263703 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 17:28:22.264869 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:28:22.264843 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee05c9c6_d540_496c_8fc3_c1cdf5c88585.slice/crio-47305bfd69d08e989e802f92138a52ac165f92dfa991fdd32fa8c1ba449390bf WatchSource:0}: Error finding container 47305bfd69d08e989e802f92138a52ac165f92dfa991fdd32fa8c1ba449390bf: Status 404 returned error can't find the container with id 47305bfd69d08e989e802f92138a52ac165f92dfa991fdd32fa8c1ba449390bf Oct 30 17:28:23.020090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:23.020054 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" event={"ID":"ee05c9c6-d540-496c-8fc3-c1cdf5c88585","Type":"ContainerStarted","Data":"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8"} Oct 30 17:28:23.020090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:23.020105 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" event={"ID":"ee05c9c6-d540-496c-8fc3-c1cdf5c88585","Type":"ContainerStarted","Data":"47305bfd69d08e989e802f92138a52ac165f92dfa991fdd32fa8c1ba449390bf"} Oct 30 17:28:23.064120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:23.064065 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podStartSLOduration=2.064052655 podStartE2EDuration="2.064052655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:23.062442234 +0000 UTC m=+930.839914190" watchObservedRunningTime="2026-10-30 17:28:23.064052655 +0000 UTC m=+930.841524609" Oct 30 17:28:23.966852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:23.966821 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:24.533865 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:28:24.533830 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:28:31.966693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:31.966660 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:31.979458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:31.979434 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:32.083248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:32.083222 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:28:34.451168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.451136 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g"] Oct 30 17:28:34.455219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.455201 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.477822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.477792 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g"] Oct 30 17:28:34.486323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.486297 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:28:34.486464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.486424 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:28:34.601719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.601685 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ntn\" (UniqueName: \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.601872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.601740 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.601872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.601807 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.703195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.703120 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.703195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.703168 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.703400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.703241 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ntn\" (UniqueName: \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.703961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.703936 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.705525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.705504 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.726699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.726678 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ntn\" (UniqueName: \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\") pod \"maas-api-revoke-sub-4ef8426c-2nm7g\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:34.769350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:34.769328 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:35.072324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:35.072296 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g"] Oct 30 17:28:35.073240 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:28:35.073205 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc87ed61_3e8b_4083_adfe_d30f38e1bbf2.slice/crio-07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc WatchSource:0}: Error finding container 07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc: Status 404 returned error can't find the container with id 07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc Oct 30 17:28:35.075553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:35.075536 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:28:35.091623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:35.091594 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" event={"ID":"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2","Type":"ContainerStarted","Data":"07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc"} Oct 30 17:28:38.012632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.012604 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 17:28:38.016550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.016532 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.037960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.037938 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-506124-openshift-default-dockercfg-xdvhh\"" Oct 30 17:28:38.053819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.053778 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 17:28:38.110529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.110499 4622 generic.go:356] "Generic (PLEG): container finished" podID="dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" containerID="6185f08d706cf8cb888cd7376667c2c552bc4fa934ebd6d384d45263d0dea1b3" exitCode=0 Oct 30 17:28:38.110680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.110540 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" event={"ID":"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2","Type":"ContainerDied","Data":"6185f08d706cf8cb888cd7376667c2c552bc4fa934ebd6d384d45263d0dea1b3"} Oct 30 17:28:38.140214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140190 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140228 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140360 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140470 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140449 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140498 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140527 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140560 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140656 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140628 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.140730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.140711 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k2th\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241421 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241456 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241510 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241551 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241576 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241619 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241667 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241689 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241826 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k2th\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.241962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.241854 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.242122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.242035 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.242122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.242080 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.242248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.242119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.242374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.242356 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.243948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.243924 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.244090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.244071 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.272019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.271983 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.272148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.272124 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k2th\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.335931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.335908 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:38.652633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.652606 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 17:28:38.653104 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:28:38.653084 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16a824b_5141_48b3_aa25_a30297c44ffa.slice/crio-3eb7412892235898ce700ea79f10972896b6acf29d410488853eee59f3ea465b WatchSource:0}: Error finding container 3eb7412892235898ce700ea79f10972896b6acf29d410488853eee59f3ea465b: Status 404 returned error can't find the container with id 3eb7412892235898ce700ea79f10972896b6acf29d410488853eee59f3ea465b Oct 30 17:28:38.655460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.655432 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:38.655526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.655493 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:38.655562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:38.655527 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:28:39.118817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.118780 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" event={"ID":"d16a824b-5141-48b3-aa25-a30297c44ffa","Type":"ContainerStarted","Data":"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370"} Oct 30 17:28:39.119246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.118835 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" event={"ID":"d16a824b-5141-48b3-aa25-a30297c44ffa","Type":"ContainerStarted","Data":"3eb7412892235898ce700ea79f10972896b6acf29d410488853eee59f3ea465b"} Oct 30 17:28:39.329670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.329646 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:39.336528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.336502 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:39.338151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.338127 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:39.338295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.338164 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:39.384528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.384417 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podStartSLOduration=2.38439747 podStartE2EDuration="2.38439747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:39.168750334 +0000 UTC m=+946.946222290" watchObservedRunningTime="2026-10-30 17:28:39.38439747 +0000 UTC m=+947.161869426" Oct 30 17:28:39.471711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.471665 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\") pod \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " Oct 30 17:28:39.472100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.472077 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\" (UniqueName: \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\") pod \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " Oct 30 17:28:39.472325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.472250 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\") pod \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\" (UID: \"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2\") " Oct 30 17:28:39.474386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.474360 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca" pod "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" (UID: "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:39.476006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.475981 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca" pod "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" (UID: "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:39.476092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.475988 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn" pod "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" (UID: "dc87ed61-3e8b-4083-adfe-d30f38e1bbf2"). InnerVolumeSpecName "kube-api-access-66ntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:39.573957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.573923 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66ntn\" (UniqueName: \"kubernetes.io/projected/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-kube-api-access-66ntn\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:28:39.573957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.573951 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:28:39.573957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:39.573962 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:28:40.125878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:40.125848 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" event={"ID":"dc87ed61-3e8b-4083-adfe-d30f38e1bbf2","Type":"ContainerDied","Data":"07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc"} Oct 30 17:28:40.125878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:40.125866 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g" Oct 30 17:28:40.126364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:40.125897 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d5f78d135fcfc7f0e7936eb2a91dace9f9fca21ab2f54eb38c9cf5337093cc" Oct 30 17:28:40.336445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:40.336408 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:40.336626 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:40.336458 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:41.336653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:41.336622 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:41.337040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:41.336669 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:42.336848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:42.336810 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:42.337354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:42.336871 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:43.337088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:43.337055 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:43.337497 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:43.337118 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:44.336708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:44.336671 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 17:28:44.336901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:44.336721 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 17:28:45.340161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:45.340122 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:45.340655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:45.340597 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:45.341350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:28:45.341332 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 17:28:54.534382 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:28:54.534339 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:29:00.332033 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.331996 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh"] Oct 30 17:29:00.332507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.332471 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" containerName="revoke-keys" Oct 30 17:29:00.332507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.332485 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" containerName="revoke-keys" Oct 30 17:29:00.332613 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.332601 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" containerName="revoke-keys" Oct 30 17:29:00.335914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.335893 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.363883 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.363840 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:29:00.364078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.363934 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:29:00.366565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.366535 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh"] Oct 30 17:29:00.382887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.381784 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.382887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.381873 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46fn\" (UniqueName: \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.382887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.381976 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.483243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.483204 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.483424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.483299 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.483424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.483356 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46fn\" (UniqueName: \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.483950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.483924 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.485735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.485710 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.506595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.506571 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46fn\" (UniqueName: \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\") pod \"maas-api-revoke-sub-f2b30573-6dlrh\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.650989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.650905 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:00.835767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.835738 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk"] Oct 30 17:29:00.839684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.839668 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.859625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.859596 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk"] Oct 30 17:29:00.887004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.886973 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.887156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.887046 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.887220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.887176 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsp7\" (UniqueName: \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.950016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.949986 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh"] Oct 30 17:29:00.950402 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:00.950382 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e36521d_025c_4964_9f41_d0b399495eb5.slice/crio-baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f WatchSource:0}: Error finding container baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f: Status 404 returned error can't find the container with id baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f Oct 30 17:29:00.988398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.988371 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.988535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.988469 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsp7\" (UniqueName: \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.988586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.988570 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.989227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.989202 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:00.990832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:00.990806 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:01.012530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.012504 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsp7\" (UniqueName: \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\") pod \"maas-api-revoke-sub-95c0bb8a-sfjbk\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:01.153720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.153688 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:01.271968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.271939 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" event={"ID":"3e36521d-025c-4964-9f41-d0b399495eb5","Type":"ContainerStarted","Data":"fb902a09274d2386a9a1e9ffaa68755accf70d11afd29586eea8154040b753fb"} Oct 30 17:29:01.272109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.271988 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" event={"ID":"3e36521d-025c-4964-9f41-d0b399495eb5","Type":"ContainerStarted","Data":"baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f"} Oct 30 17:29:01.313618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.313560 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" podStartSLOduration=1.313545279 podStartE2EDuration="1.313545279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:01.312722596 +0000 UTC m=+969.090194551" watchObservedRunningTime="2026-10-30 17:29:01.313545279 +0000 UTC m=+969.091017235" Oct 30 17:29:01.454769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:01.454734 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk"] Oct 30 17:29:01.455162 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:01.455138 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58dbbc5_6bfe_4d2a_a9dc_b37d61ed4e82.slice/crio-ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01 WatchSource:0}: Error finding container ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01: Status 404 returned error can't find the container with id ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01 Oct 30 17:29:02.279150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:02.279114 4622 generic.go:356] "Generic (PLEG): container finished" podID="3e36521d-025c-4964-9f41-d0b399495eb5" containerID="fb902a09274d2386a9a1e9ffaa68755accf70d11afd29586eea8154040b753fb" exitCode=0 Oct 30 17:29:02.279347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:02.279201 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" event={"ID":"3e36521d-025c-4964-9f41-d0b399495eb5","Type":"ContainerDied","Data":"fb902a09274d2386a9a1e9ffaa68755accf70d11afd29586eea8154040b753fb"} Oct 30 17:29:02.280732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:02.280708 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" event={"ID":"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82","Type":"ContainerStarted","Data":"379bd115ae8ca25e1333c5cb7495018a6bd1fac427d3ee645bed95bd2e1cec0a"} Oct 30 17:29:02.280847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:02.280752 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" event={"ID":"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82","Type":"ContainerStarted","Data":"ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01"} Oct 30 17:29:02.363510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:02.363461 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" podStartSLOduration=2.363448095 podStartE2EDuration="2.363448095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:02.361759443 +0000 UTC m=+970.139231410" watchObservedRunningTime="2026-10-30 17:29:02.363448095 +0000 UTC m=+970.140920050" Oct 30 17:29:03.287681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.287643 4622 generic.go:356] "Generic (PLEG): container finished" podID="f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" containerID="379bd115ae8ca25e1333c5cb7495018a6bd1fac427d3ee645bed95bd2e1cec0a" exitCode=0 Oct 30 17:29:03.288253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.287685 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" event={"ID":"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82","Type":"ContainerDied","Data":"379bd115ae8ca25e1333c5cb7495018a6bd1fac427d3ee645bed95bd2e1cec0a"} Oct 30 17:29:03.441715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.441692 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:03.511949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.511902 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\" (UniqueName: \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\") pod \"3e36521d-025c-4964-9f41-d0b399495eb5\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " Oct 30 17:29:03.512150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.512042 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\") pod \"3e36521d-025c-4964-9f41-d0b399495eb5\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " Oct 30 17:29:03.512244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.512157 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\") pod \"3e36521d-025c-4964-9f41-d0b399495eb5\" (UID: \"3e36521d-025c-4964-9f41-d0b399495eb5\") " Oct 30 17:29:03.512587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.512535 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca" pod "3e36521d-025c-4964-9f41-d0b399495eb5" (UID: "3e36521d-025c-4964-9f41-d0b399495eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:03.512966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.512934 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:03.514766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.514741 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn" pod "3e36521d-025c-4964-9f41-d0b399495eb5" (UID: "3e36521d-025c-4964-9f41-d0b399495eb5"). InnerVolumeSpecName "kube-api-access-j46fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:03.515017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.514985 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca" pod "3e36521d-025c-4964-9f41-d0b399495eb5" (UID: "3e36521d-025c-4964-9f41-d0b399495eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:03.614168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.614093 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46fn\" (UniqueName: \"kubernetes.io/projected/3e36521d-025c-4964-9f41-d0b399495eb5-kube-api-access-j46fn\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:03.614168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:03.614118 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e36521d-025c-4964-9f41-d0b399495eb5-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:04.295949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.295921 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" Oct 30 17:29:04.296402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.295920 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh" event={"ID":"3e36521d-025c-4964-9f41-d0b399495eb5","Type":"ContainerDied","Data":"baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f"} Oct 30 17:29:04.296402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.296067 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa61783c9b8d69e9ca8d28a611009ded5dcd99b96eb7badaaf9a0215055181f" Oct 30 17:29:04.438462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.438437 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:04.522063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.522037 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\") pod \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " Oct 30 17:29:04.522249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.522147 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\" (UniqueName: \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\") pod \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " Oct 30 17:29:04.522249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.522185 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\") pod \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\" (UID: \"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82\") " Oct 30 17:29:04.522576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.522546 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca" pod "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" (UID: "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:04.523970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.523948 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca" pod "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" (UID: "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:04.524240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.524216 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7" pod "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" (UID: "f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82"). InnerVolumeSpecName "kube-api-access-dqsp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:04.623559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.623483 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:04.623559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.623508 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsp7\" (UniqueName: \"kubernetes.io/projected/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-kube-api-access-dqsp7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:04.623559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:04.623519 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:29:05.302519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:05.302493 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" Oct 30 17:29:05.302912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:05.302531 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk" event={"ID":"f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82","Type":"ContainerDied","Data":"ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01"} Oct 30 17:29:05.302912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:05.302580 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec35361edf3847560c0dfc1f70fc87f87261dc3689afb099fb9ff16a59ea2e01" Oct 30 17:29:07.470842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.470808 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:29:07.471479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471459 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" containerName="revoke-keys" Oct 30 17:29:07.471577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471481 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" containerName="revoke-keys" Oct 30 17:29:07.471577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471511 4622 cpu_manager.go:414] "containerMap: removing container" podUID="3e36521d-025c-4964-9f41-d0b399495eb5" containerName="revoke-keys" Oct 30 17:29:07.471577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471520 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e36521d-025c-4964-9f41-d0b399495eb5" containerName="revoke-keys" Oct 30 17:29:07.471732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471649 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" containerName="revoke-keys" Oct 30 17:29:07.471732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.471667 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e36521d-025c-4964-9f41-d0b399495eb5" containerName="revoke-keys" Oct 30 17:29:07.475433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.475414 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.494143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.494117 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-27567a-openshift-default-dockercfg-cnckb\"" Oct 30 17:29:07.523431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.523399 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:29:07.555954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.555928 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt46x\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.555973 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556004 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556164 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556229 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556307 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556340 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556360 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.556450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.556436 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.657706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657668 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.657879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657754 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.657879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657807 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt46x\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.657879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657860 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657897 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657939 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.657993 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658047 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658087 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658103 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658455 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658373 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658598 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658726 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.658831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.658803 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.660988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.660851 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.660988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.660897 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.683134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.683098 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.686327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.686301 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt46x\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.709595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.709568 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:29:07.714856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.714835 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.731896 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.731875 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-455086-openshift-default-dockercfg-98bfj\"" Oct 30 17:29:07.735191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.735168 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:29:07.758757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.758723 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.758901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.758784 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.758901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.758836 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprcj\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759045 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759022 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759071 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759179 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759236 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759216 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759247 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.759336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.759295 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.789151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.789122 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867456 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867518 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867549 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprcj\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867608 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867647 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867715 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867741 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867773 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.867809 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.875434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.868983 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.880016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.878782 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.880016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.879394 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.880016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.879643 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.880016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.879883 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.880415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.880119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.883089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.883060 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.892100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.892074 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:07.901856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:07.901811 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprcj\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:08.029145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.029075 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:08.118947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.118915 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:29:08.119062 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:08.119011 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e6c61_d33f_4392_be49_bd7b6ed21b91.slice/crio-9d08e198a6bb1fb0af9607679c100f368dbe523ef26cf6496fecd1637da00457 WatchSource:0}: Error finding container 9d08e198a6bb1fb0af9607679c100f368dbe523ef26cf6496fecd1637da00457: Status 404 returned error can't find the container with id 9d08e198a6bb1fb0af9607679c100f368dbe523ef26cf6496fecd1637da00457 Oct 30 17:29:08.121393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.121364 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.121477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.121424 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.121477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.121454 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.321441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.321402 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" event={"ID":"7f5e6c61-d33f-4392-be49-bd7b6ed21b91","Type":"ContainerStarted","Data":"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2"} Oct 30 17:29:08.321610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.321456 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" event={"ID":"7f5e6c61-d33f-4392-be49-bd7b6ed21b91","Type":"ContainerStarted","Data":"9d08e198a6bb1fb0af9607679c100f368dbe523ef26cf6496fecd1637da00457"} Oct 30 17:29:08.342957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.342926 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:29:08.346749 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:08.346723 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479bdec4_2eb5_450f_ab89_e4c903f001a6.slice/crio-5b85277b998ae2f756f978a5fcfd455ea74037a4499b818cd096f6a86756923f WatchSource:0}: Error finding container 5b85277b998ae2f756f978a5fcfd455ea74037a4499b818cd096f6a86756923f: Status 404 returned error can't find the container with id 5b85277b998ae2f756f978a5fcfd455ea74037a4499b818cd096f6a86756923f Oct 30 17:29:08.349716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.349679 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.349830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.349764 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.349830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.349808 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:08.375214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.375001 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podStartSLOduration=1.374982583 podStartE2EDuration="1.374982583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:08.370828748 +0000 UTC m=+976.148300705" watchObservedRunningTime="2026-10-30 17:29:08.374982583 +0000 UTC m=+976.152454539" Oct 30 17:29:08.789664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:08.789631 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:09.328811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:09.328710 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" event={"ID":"479bdec4-2eb5-450f-ab89-e4c903f001a6","Type":"ContainerStarted","Data":"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a"} Oct 30 17:29:09.328811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:09.328772 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" event={"ID":"479bdec4-2eb5-450f-ab89-e4c903f001a6","Type":"ContainerStarted","Data":"5b85277b998ae2f756f978a5fcfd455ea74037a4499b818cd096f6a86756923f"} Oct 30 17:29:09.407726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:09.407554 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podStartSLOduration=2.407539927 podStartE2EDuration="2.407539927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:09.405958068 +0000 UTC m=+977.183430023" watchObservedRunningTime="2026-10-30 17:29:09.407539927 +0000 UTC m=+977.185011882" Oct 30 17:29:09.792804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:09.792661 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:09.792804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:09.792717 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:10.029678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:10.029647 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:10.031542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:10.031512 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:10.031694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:10.031560 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:10.790102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:10.790067 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:10.790415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:10.790382 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:11.030110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:11.030078 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:11.030486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:11.030141 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:11.796858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:11.789706 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:11.796858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:11.789777 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:12.030341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.030312 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:12.030814 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.030790 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:12.648286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.645156 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:29:12.673489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.671519 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.698644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.696522 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:29:12.727562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727532 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727573 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727599 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727626 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727653 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsxt\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727683 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727701 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727731 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.727930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.727756 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.792286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.790651 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:12.792286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.790706 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:12.828640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828606 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.828831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828647 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.828831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828682 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.828831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828723 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.828831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828762 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhsxt\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.828831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828805 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828834 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828881 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828940 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.828960 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.829377 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.829898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.829870 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.830124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.830103 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.830669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.830645 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.831364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.831345 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.832243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.832223 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.861231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.861164 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhsxt\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:12.862690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:12.862666 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-rzcv5\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:13.015761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.015647 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:13.030010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.029983 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:13.030089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.030038 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:13.404837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.404803 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:29:13.407812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.407776 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:13.407918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.407860 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:13.407918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.407903 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:13.797316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.791695 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:13.797316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:13.791772 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:14.033282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.029596 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:14.033282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.029654 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:14.404895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.403820 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" event={"ID":"af47d08e-6284-4f32-88d2-9b7bdd32c756","Type":"ContainerStarted","Data":"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946"} Oct 30 17:29:14.404895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.403888 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" event={"ID":"af47d08e-6284-4f32-88d2-9b7bdd32c756","Type":"ContainerStarted","Data":"e523e0a61fa36debf4c3cd92e5fa3e6da0b4d262003c26fa6aae469386d1c5ce"} Oct 30 17:29:14.474634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.474577 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podStartSLOduration=2.474558813 podStartE2EDuration="2.474558813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:14.470605759 +0000 UTC m=+982.248077714" watchObservedRunningTime="2026-10-30 17:29:14.474558813 +0000 UTC m=+982.252030769" Oct 30 17:29:14.790009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.789968 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 17:29:14.790206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:14.790038 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 17:29:15.016530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.016496 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:15.018166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.018139 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:15.018306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.018193 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:15.030152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.030120 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 17:29:15.030311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.030168 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 17:29:15.793684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.793657 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:15.794073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.794052 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:15.794647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:15.794625 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:29:16.017039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:16.017002 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:16.017239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:16.017055 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:16.033407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:16.033379 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:16.033746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:16.033727 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:16.034584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:16.034564 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:29:17.016165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:17.016126 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:17.016583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:17.016191 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:18.016459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:18.016426 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:18.016932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:18.016477 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:19.016517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:19.016480 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:19.016899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:19.016548 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:20.016017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:20.015982 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:20.016190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:20.016040 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:21.016093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:21.016063 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 17:29:21.016469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:21.016113 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 17:29:22.142160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.142125 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:22.142700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.142545 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:22.143828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.143807 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:29:22.515542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.515470 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:29:22.519437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.519419 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.549202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.549178 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:29:22.571489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.571462 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:29:22.575347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.575328 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.590381 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.590363 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-db98b0-openshift-default-dockercfg-qvj6k\"" Oct 30 17:29:22.599162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.599139 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:29:22.709327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709293 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.709504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709339 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.709504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709427 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.709504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709452 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.709732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709556 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v94d\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.709732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709668 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.709732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709716 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.709897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709757 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vvq\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.709897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709832 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.709897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709864 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.710048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709909 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.710048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.709954 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.710048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710029 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.710170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710060 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.710170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710078 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.710170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710099 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.710170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710155 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.710383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.710189 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812119 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812189 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812224 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812317 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v94d\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812357 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812400 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812451 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vvq\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812495 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812530 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812580 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812631 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812699 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812728 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812753 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812788 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812813 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812839 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.812900 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.813565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.813439 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.815001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.814975 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.815091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.815012 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.815519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.815491 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.815869 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.815725 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.816291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.816197 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.817250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.816697 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.817250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.816947 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.817250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.817127 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.817513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.817477 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.822214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.820624 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.822214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.820644 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.823114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.823055 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.823819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.823761 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.839721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.839689 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.839925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.839899 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vvq\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.839967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.839920 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v94d\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:22.841862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.841835 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:22.889408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:22.889367 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:23.132934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.132836 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:23.242278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.242221 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:29:23.256952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.254954 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:23.256952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.255046 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:23.256952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.255091 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:23.492650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.492550 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" event={"ID":"78329738-7ff8-4f4f-9b71-51cc9abc18cb","Type":"ContainerStarted","Data":"3fc31412853a4851982f7f61adfeccee3a718deef2f53138a14a2a6734360420"} Oct 30 17:29:23.522553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.522087 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:29:23.534681 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:23.534651 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d7627f_21fc_4cdf_b5d1_dcf817098b00.slice/crio-1cff6686889a857af7edb783ca32415fc1e129249b0b2203a186a01ab2cb902b WatchSource:0}: Error finding container 1cff6686889a857af7edb783ca32415fc1e129249b0b2203a186a01ab2cb902b: Status 404 returned error can't find the container with id 1cff6686889a857af7edb783ca32415fc1e129249b0b2203a186a01ab2cb902b Oct 30 17:29:23.537382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.537345 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:23.537483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.537434 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:23.537546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:23.537482 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:24.498857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.498818 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" event={"ID":"78329738-7ff8-4f4f-9b71-51cc9abc18cb","Type":"ContainerStarted","Data":"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc"} Oct 30 17:29:24.500543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.500515 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" event={"ID":"37d7627f-21fc-4cdf-b5d1-dcf817098b00","Type":"ContainerStarted","Data":"053b6c01be206a7a2ec3ec20306b7a923d637a3fd9a4912cb24d80ac1a4797ec"} Oct 30 17:29:24.500642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.500561 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" event={"ID":"37d7627f-21fc-4cdf-b5d1-dcf817098b00","Type":"ContainerStarted","Data":"1cff6686889a857af7edb783ca32415fc1e129249b0b2203a186a01ab2cb902b"} Oct 30 17:29:24.533625 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:29:24.533596 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:29:24.581052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.580999 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podStartSLOduration=2.580986019 podStartE2EDuration="2.580986019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:24.566783729 +0000 UTC m=+992.344255683" watchObservedRunningTime="2026-10-30 17:29:24.580986019 +0000 UTC m=+992.358457974" Oct 30 17:29:24.612527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.612484 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podStartSLOduration=2.612469801 podStartE2EDuration="2.612469801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:24.611662249 +0000 UTC m=+992.389134203" watchObservedRunningTime="2026-10-30 17:29:24.612469801 +0000 UTC m=+992.389941755" Oct 30 17:29:24.901356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.894597 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:24.901640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.901401 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:24.901640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:24.901460 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:25.134152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:25.134082 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:25.135817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:25.135794 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:25.135914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:25.135841 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:25.892313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:25.891679 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:25.892313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:25.891740 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:26.134212 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:26.134179 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:26.134381 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:26.134232 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:26.891764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:26.891724 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:26.892027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:26.891998 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:27.134245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:27.134213 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:27.134706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:27.134288 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:27.891140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:27.891101 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:27.891481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:27.891452 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:28.134510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:28.134481 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:28.134974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:28.134945 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:28.890791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:28.890743 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:28.890969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:28.890811 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:29.134347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:29.134310 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:29.134494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:29.134362 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:29.890634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:29.890596 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 17:29:29.891224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:29.890649 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 17:29:30.133545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:30.133512 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 17:29:30.133709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:30.133563 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 17:29:30.894886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:30.894851 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:30.895380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:30.895304 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:30.896283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:30.896244 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:29:31.144049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:31.144016 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:31.145032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:31.144956 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:31.145612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:31.145583 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:29:52.931496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:52.931422 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:29:52.936643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:52.936627 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:52.956007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:52.955984 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:29:52.965004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:52.964983 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0e7403-openshift-default-dockercfg-z9pr5\"" Oct 30 17:29:53.032542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032511 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032551 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032580 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032697 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032728 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032861 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032943 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032911 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.032995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.032980 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4sw\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.033044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.033005 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.070884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.070854 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:29:53.074787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.074768 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.092603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.092583 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-306aed-openshift-default-dockercfg-rbbs2\"" Oct 30 17:29:53.099304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.099280 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:29:53.133711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133681 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.133847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133728 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.133847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133768 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.133847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133812 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133845 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133940 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.133972 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134015 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4sw\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134049 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134142 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134201 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134439 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134540 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.134678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.134551 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.136385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.136360 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.136590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.136574 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.157227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.157209 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.157675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.157658 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4sw\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.235674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235605 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.235674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235641 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.235850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235759 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.235850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235798 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbh7\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.235968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235910 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.235968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.235956 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.236108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.236090 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.236154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.236129 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.236204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.236154 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.252537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.252514 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:53.337087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337057 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337099 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbh7\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337132 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337158 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337237 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337345 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337375 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337484 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337484 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337514 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337689 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337785 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.337877 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.337836 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.338358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.338287 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.340223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.339882 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.340223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.340176 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.360594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.360565 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.360723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.360702 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbh7\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.390793 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.390766 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:53.583461 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:53.583421 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b4b9c_19ad_4691_a2ff_9b2b315b15d0.slice/crio-de0024a3eb75a9d1c42651f5fdf0dbc7078cb825be0ec5f8452b538531ff6e10 WatchSource:0}: Error finding container de0024a3eb75a9d1c42651f5fdf0dbc7078cb825be0ec5f8452b538531ff6e10: Status 404 returned error can't find the container with id de0024a3eb75a9d1c42651f5fdf0dbc7078cb825be0ec5f8452b538531ff6e10 Oct 30 17:29:53.583629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.583607 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:29:53.585875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.585844 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.585955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.585906 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.585955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.585931 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.694288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.694244 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:29:53.698954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.698919 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" event={"ID":"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0","Type":"ContainerStarted","Data":"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97"} Oct 30 17:29:53.699111 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.698980 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" event={"ID":"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0","Type":"ContainerStarted","Data":"de0024a3eb75a9d1c42651f5fdf0dbc7078cb825be0ec5f8452b538531ff6e10"} Oct 30 17:29:53.707784 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:29:53.707755 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd309e61b_f5f9_41c7_9ffa_0a8ba650df11.slice/crio-0d48e5b341db74ae8fa0950c55d5a645eb7ef9ff33f650857e7346f55208d9dc WatchSource:0}: Error finding container 0d48e5b341db74ae8fa0950c55d5a645eb7ef9ff33f650857e7346f55208d9dc: Status 404 returned error can't find the container with id 0d48e5b341db74ae8fa0950c55d5a645eb7ef9ff33f650857e7346f55208d9dc Oct 30 17:29:53.710838 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.710804 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.710917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.710887 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.710963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.710927 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:29:53.745845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:53.745789 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podStartSLOduration=1.745768745 podStartE2EDuration="1.745768745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:53.742761432 +0000 UTC m=+1021.520233389" watchObservedRunningTime="2026-10-30 17:29:53.745768745 +0000 UTC m=+1021.523240702" Oct 30 17:29:54.257286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.256558 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:29:54.257286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.256777 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:54.257286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.256843 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:54.534893 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:29:54.534857 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:29:54.710034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.710005 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" event={"ID":"d309e61b-f5f9-41c7-9ffa-0a8ba650df11","Type":"ContainerStarted","Data":"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446"} Oct 30 17:29:54.710203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.710047 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" event={"ID":"d309e61b-f5f9-41c7-9ffa-0a8ba650df11","Type":"ContainerStarted","Data":"0d48e5b341db74ae8fa0950c55d5a645eb7ef9ff33f650857e7346f55208d9dc"} Oct 30 17:29:54.773286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:54.772942 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podStartSLOduration=1.772924151 podStartE2EDuration="1.772924151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:54.771568718 +0000 UTC m=+1022.549040674" watchObservedRunningTime="2026-10-30 17:29:54.772924151 +0000 UTC m=+1022.550396108" Oct 30 17:29:55.261287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:55.253749 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:55.261287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:55.253835 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:55.392304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:55.391593 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:29:55.393576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:55.393548 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:55.393666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:55.393599 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 17:29:56.256674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:56.256632 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:56.256963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:56.256932 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:56.398314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:56.397638 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:56.398314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:56.397697 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 17:29:57.256709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:57.256671 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:57.257010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:57.256983 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:57.392367 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:57.391931 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:57.392367 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:57.391994 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 17:29:58.259287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:58.257343 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:58.259287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:58.257405 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:58.392283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:58.391763 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:58.392283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:58.391844 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 17:29:59.261285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:59.256628 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 17:29:59.261285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:59.256688 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 17:29:59.391760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:59.391717 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 17:29:59.391978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:29:59.391784 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 17:30:00.178282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.178241 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:00.182978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.182954 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.183453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.183428 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:00.187974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.187954 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.217215 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.217188 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:30:00.264789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.264746 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:00.269992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.269770 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:00.275460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.275432 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:00.279869 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.279830 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxqq\" (UniqueName: \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.280191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.280166 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmn5\" (UniqueName: \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.280456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.280435 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.280700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.280670 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.282184 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.281451 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:00.282184 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.281560 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.382483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382447 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.382676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382501 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.382676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382659 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.382843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382822 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxqq\" (UniqueName: \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.382935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382915 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmn5\" (UniqueName: \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.382997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.382982 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gdn\" (UniqueName: \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.383385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.383362 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.383541 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.383522 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.399160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.399140 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:30:00.400516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.400495 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:30:00.402013 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.401991 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:30:00.413141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.413119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxqq\" (UniqueName: \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\") pod \"maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.413245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.413171 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmn5\" (UniqueName: \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\") pod \"maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.483923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.483840 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.484153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.484125 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gdn\" (UniqueName: \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.484557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.484533 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.502662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.502635 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:00.528745 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.528711 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:00.532956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.529554 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gdn\" (UniqueName: \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.613406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.613120 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:00.765415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.765170 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:30:00.767081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.766764 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:30:00.769776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.769725 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:30:00.840333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.840312 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:00.844579 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:00.844547 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158a45e1_a395_4f6c_9c76_12c4c71896f3.slice/crio-280f00d067dd869f5338044181a5f79288271baf36f36f8b94010d893d48360c WatchSource:0}: Error finding container 280f00d067dd869f5338044181a5f79288271baf36f36f8b94010d893d48360c: Status 404 returned error can't find the container with id 280f00d067dd869f5338044181a5f79288271baf36f36f8b94010d893d48360c Oct 30 17:30:00.870176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.870157 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:00.876698 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:00.876663 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e32b91_04b3_40ca_97fe_1c3d1034659c.slice/crio-a4cad26a1a6750cb34af1613c5cc4fc7f0a8a13ed1140eed278cf23147ddf2e3 WatchSource:0}: Error finding container a4cad26a1a6750cb34af1613c5cc4fc7f0a8a13ed1140eed278cf23147ddf2e3: Status 404 returned error can't find the container with id a4cad26a1a6750cb34af1613c5cc4fc7f0a8a13ed1140eed278cf23147ddf2e3 Oct 30 17:30:00.950176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:00.950150 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:00.950344 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:00.950193 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f11800_3201_4095_bbc9_b3df311a6f21.slice/crio-352dd06c4292adc0e316e3491a5a534dd9b77e456d7a17fbc0da097d1167d1e1 WatchSource:0}: Error finding container 352dd06c4292adc0e316e3491a5a534dd9b77e456d7a17fbc0da097d1167d1e1: Status 404 returned error can't find the container with id 352dd06c4292adc0e316e3491a5a534dd9b77e456d7a17fbc0da097d1167d1e1 Oct 30 17:30:01.771771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.771730 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerStarted","Data":"f701a27fdce0da35528bd3d387f7eb091129e64dbe4a6d3d5cb89c6497663137"} Oct 30 17:30:01.772340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.771838 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerStarted","Data":"352dd06c4292adc0e316e3491a5a534dd9b77e456d7a17fbc0da097d1167d1e1"} Oct 30 17:30:01.773765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.773735 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerStarted","Data":"ab3fb9da5253bb9a795310b5e8ef1cb7a87ce781e9140f74a6b464306ef4857b"} Oct 30 17:30:01.773895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.773791 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerStarted","Data":"280f00d067dd869f5338044181a5f79288271baf36f36f8b94010d893d48360c"} Oct 30 17:30:01.776698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.776667 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerStarted","Data":"383823ae6235127d10a2aa0193ebc1a6c42eb2f552d01c3dfd7f9743877d7c7b"} Oct 30 17:30:01.776826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.776717 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerStarted","Data":"a4cad26a1a6750cb34af1613c5cc4fc7f0a8a13ed1140eed278cf23147ddf2e3"} Oct 30 17:30:01.819747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.819682 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" podStartSLOduration=1.819663715 podStartE2EDuration="1.819663715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:01.815532654 +0000 UTC m=+1029.593004608" watchObservedRunningTime="2026-10-30 17:30:01.819663715 +0000 UTC m=+1029.597135672" Oct 30 17:30:01.862505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.860199 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" podStartSLOduration=1.860184474 podStartE2EDuration="1.860184474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:01.857612287 +0000 UTC m=+1029.635084243" watchObservedRunningTime="2026-10-30 17:30:01.860184474 +0000 UTC m=+1029.637656430" Oct 30 17:30:01.899949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:01.899895 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" podStartSLOduration=1.899880649 podStartE2EDuration="1.899880649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:01.897255225 +0000 UTC m=+1029.674727182" watchObservedRunningTime="2026-10-30 17:30:01.899880649 +0000 UTC m=+1029.677352602" Oct 30 17:30:02.784929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.784887 4622 generic.go:356] "Generic (PLEG): container finished" podID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerID="ab3fb9da5253bb9a795310b5e8ef1cb7a87ce781e9140f74a6b464306ef4857b" exitCode=60 Oct 30 17:30:02.785460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.785022 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerDied","Data":"ab3fb9da5253bb9a795310b5e8ef1cb7a87ce781e9140f74a6b464306ef4857b"} Oct 30 17:30:02.785594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.785483 4622 scope.go:122] "RemoveContainer" containerID="ab3fb9da5253bb9a795310b5e8ef1cb7a87ce781e9140f74a6b464306ef4857b" Oct 30 17:30:02.789845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.789817 4622 generic.go:356] "Generic (PLEG): container finished" podID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerID="383823ae6235127d10a2aa0193ebc1a6c42eb2f552d01c3dfd7f9743877d7c7b" exitCode=60 Oct 30 17:30:02.789967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.789899 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerDied","Data":"383823ae6235127d10a2aa0193ebc1a6c42eb2f552d01c3dfd7f9743877d7c7b"} Oct 30 17:30:02.790323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.790232 4622 scope.go:122] "RemoveContainer" containerID="383823ae6235127d10a2aa0193ebc1a6c42eb2f552d01c3dfd7f9743877d7c7b" Oct 30 17:30:02.804311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.800079 4622 generic.go:356] "Generic (PLEG): container finished" podID="82f11800-3201-4095-bbc9-b3df311a6f21" containerID="f701a27fdce0da35528bd3d387f7eb091129e64dbe4a6d3d5cb89c6497663137" exitCode=60 Oct 30 17:30:02.804311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.801236 4622 scope.go:122] "RemoveContainer" containerID="f701a27fdce0da35528bd3d387f7eb091129e64dbe4a6d3d5cb89c6497663137" Oct 30 17:30:02.804311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:02.801579 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerDied","Data":"f701a27fdce0da35528bd3d387f7eb091129e64dbe4a6d3d5cb89c6497663137"} Oct 30 17:30:03.809387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.809356 4622 generic.go:356] "Generic (PLEG): container finished" podID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerID="220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73" exitCode=60 Oct 30 17:30:03.809801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.809485 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerDied","Data":"220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73"} Oct 30 17:30:03.809801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.809540 4622 scope.go:122] "RemoveContainer" containerID="ab3fb9da5253bb9a795310b5e8ef1cb7a87ce781e9140f74a6b464306ef4857b" Oct 30 17:30:03.809929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.809814 4622 scope.go:122] "RemoveContainer" containerID="220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73" Oct 30 17:30:03.810144 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:03.810124 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp_odh-ai-gateway-infra(158a45e1-a395-4f6c-9c76-12c4c71896f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" Oct 30 17:30:03.812324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.812304 4622 generic.go:356] "Generic (PLEG): container finished" podID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerID="058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0" exitCode=60 Oct 30 17:30:03.812425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.812391 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerDied","Data":"058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0"} Oct 30 17:30:03.812679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.812665 4622 scope.go:122] "RemoveContainer" containerID="058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0" Oct 30 17:30:03.812971 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:03.812948 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj_odh-ai-gateway-infra(53e32b91-04b3-40ca-97fe-1c3d1034659c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" Oct 30 17:30:03.814385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.814362 4622 generic.go:356] "Generic (PLEG): container finished" podID="82f11800-3201-4095-bbc9-b3df311a6f21" containerID="f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56" exitCode=60 Oct 30 17:30:03.814493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.814425 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerDied","Data":"f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56"} Oct 30 17:30:03.814693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.814677 4622 scope.go:122] "RemoveContainer" containerID="f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56" Oct 30 17:30:03.814939 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:03.814920 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j_odh-ai-gateway-infra(82f11800-3201-4095-bbc9-b3df311a6f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" Oct 30 17:30:03.880023 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.880003 4622 scope.go:122] "RemoveContainer" containerID="383823ae6235127d10a2aa0193ebc1a6c42eb2f552d01c3dfd7f9743877d7c7b" Oct 30 17:30:03.893149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:03.893129 4622 scope.go:122] "RemoveContainer" containerID="f701a27fdce0da35528bd3d387f7eb091129e64dbe4a6d3d5cb89c6497663137" Oct 30 17:30:04.832038 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:04.832003 4622 scope.go:122] "RemoveContainer" containerID="f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56" Oct 30 17:30:04.832555 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:04.832380 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j_odh-ai-gateway-infra(82f11800-3201-4095-bbc9-b3df311a6f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" Oct 30 17:30:04.834139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:04.834117 4622 scope.go:122] "RemoveContainer" containerID="220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73" Oct 30 17:30:04.834449 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:04.834416 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp_odh-ai-gateway-infra(158a45e1-a395-4f6c-9c76-12c4c71896f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" Oct 30 17:30:04.836209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:04.836187 4622 scope.go:122] "RemoveContainer" containerID="058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0" Oct 30 17:30:04.836626 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:04.836570 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj_odh-ai-gateway-infra(53e32b91-04b3-40ca-97fe-1c3d1034659c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" Oct 30 17:30:07.710704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.710671 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:30:07.715926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.715906 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.758224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.758188 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:30:07.775954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.775902 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776251 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776464 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776663 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfm8j\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776708 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776768 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.776813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776796 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.777044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776840 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.777044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.776868 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878142 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878189 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfm8j\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878212 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878293 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878326 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878373 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878699 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878742 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878809 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.878935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878815 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.879014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.878997 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.879076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.879052 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.879130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.879056 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.879130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.879087 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.880971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.880947 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.881125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.881108 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.901864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.901822 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:07.902011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:07.901928 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfm8j\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:08.056453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.056416 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:08.381542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.381514 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:30:08.381655 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:08.381518 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887b5174_c727_4d7f_bf67_bedbd15eb7cd.slice/crio-2dbab7e0698c391a767dd9bafb9e1f0c978614b157439f9623f9836a0ebc863c WatchSource:0}: Error finding container 2dbab7e0698c391a767dd9bafb9e1f0c978614b157439f9623f9836a0ebc863c: Status 404 returned error can't find the container with id 2dbab7e0698c391a767dd9bafb9e1f0c978614b157439f9623f9836a0ebc863c Oct 30 17:30:08.383848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.383818 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:08.383912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.383880 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:08.383912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.383908 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:08.868949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.868898 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" event={"ID":"887b5174-c727-4d7f-bf67-bedbd15eb7cd","Type":"ContainerStarted","Data":"7c87b1df0f61ac2e026f62dda7d6f53fcf7325d29a3753a90634888eec48e8c1"} Oct 30 17:30:08.869529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.868975 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" event={"ID":"887b5174-c727-4d7f-bf67-bedbd15eb7cd","Type":"ContainerStarted","Data":"2dbab7e0698c391a767dd9bafb9e1f0c978614b157439f9623f9836a0ebc863c"} Oct 30 17:30:08.957673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:08.957581 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podStartSLOduration=1.957558428 podStartE2EDuration="1.957558428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:08.930386643 +0000 UTC m=+1036.707858601" watchObservedRunningTime="2026-10-30 17:30:08.957558428 +0000 UTC m=+1036.735030385" Oct 30 17:30:09.065288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:09.062321 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:09.065492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:09.065371 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:09.065492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:09.065440 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:10.057691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:10.057653 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:10.058413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:10.058380 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:11.065289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:11.062683 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:11.065289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:11.062756 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:12.057817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:12.057781 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:12.058179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:12.058147 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:13.060719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:13.060660 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:13.061344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:13.061311 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:14.062286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:14.061671 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:14.062286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:14.061751 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:15.057170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:15.057140 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 17:30:15.057353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:15.057196 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 17:30:16.062989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:16.062951 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:16.064358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:16.064336 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:16.065377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:16.065356 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:30:17.367536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:17.367502 4622 scope.go:122] "RemoveContainer" containerID="220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73" Oct 30 17:30:17.368018 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:17.367825 4622 scope.go:122] "RemoveContainer" containerID="f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56" Oct 30 17:30:17.939737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:17.939645 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerStarted","Data":"79506911e7b61e4099fc0157a31dc212f2ae19a5cbc7eb633aaa400275dbface"} Oct 30 17:30:17.941710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:17.941677 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerStarted","Data":"5913eb93ca7c1c995873d1e46641021167fb1feb41980cc84fb88c710000882b"} Oct 30 17:30:18.366739 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.366713 4622 scope.go:122] "RemoveContainer" containerID="058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0" Oct 30 17:30:18.949257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.949177 4622 generic.go:356] "Generic (PLEG): container finished" podID="82f11800-3201-4095-bbc9-b3df311a6f21" containerID="79506911e7b61e4099fc0157a31dc212f2ae19a5cbc7eb633aaa400275dbface" exitCode=60 Oct 30 17:30:18.949257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.949248 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerDied","Data":"79506911e7b61e4099fc0157a31dc212f2ae19a5cbc7eb633aaa400275dbface"} Oct 30 17:30:18.949810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.949322 4622 scope.go:122] "RemoveContainer" containerID="f96f3caacc2e1c341e302e21089cbcbc4686ab01b7caab631a4abfb931626d56" Oct 30 17:30:18.949810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.949746 4622 scope.go:122] "RemoveContainer" containerID="79506911e7b61e4099fc0157a31dc212f2ae19a5cbc7eb633aaa400275dbface" Oct 30 17:30:18.950103 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:18.950073 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j_odh-ai-gateway-infra(82f11800-3201-4095-bbc9-b3df311a6f21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" Oct 30 17:30:18.951258 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.951238 4622 generic.go:356] "Generic (PLEG): container finished" podID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerID="5913eb93ca7c1c995873d1e46641021167fb1feb41980cc84fb88c710000882b" exitCode=60 Oct 30 17:30:18.951366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.951300 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerDied","Data":"5913eb93ca7c1c995873d1e46641021167fb1feb41980cc84fb88c710000882b"} Oct 30 17:30:18.951636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.951615 4622 scope.go:122] "RemoveContainer" containerID="5913eb93ca7c1c995873d1e46641021167fb1feb41980cc84fb88c710000882b" Oct 30 17:30:18.951901 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:18.951883 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp_odh-ai-gateway-infra(158a45e1-a395-4f6c-9c76-12c4c71896f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" Oct 30 17:30:18.953118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.953098 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerStarted","Data":"d6f709fcc14f4c6cccf022c3f6125a7c41689d23d3ae6aa02c6fb943692c3868"} Oct 30 17:30:18.961340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.961323 4622 scope.go:122] "RemoveContainer" containerID="220bc9a085fe33be14cd0f67a1b6a81de5892d9e0a7b6a5f4046a8d19b28cc73" Oct 30 17:30:18.998744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:18.998713 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:19.040077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:19.040044 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:19.962149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:19.962117 4622 generic.go:356] "Generic (PLEG): container finished" podID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerID="d6f709fcc14f4c6cccf022c3f6125a7c41689d23d3ae6aa02c6fb943692c3868" exitCode=60 Oct 30 17:30:19.962610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:19.962185 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerDied","Data":"d6f709fcc14f4c6cccf022c3f6125a7c41689d23d3ae6aa02c6fb943692c3868"} Oct 30 17:30:19.962610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:19.962242 4622 scope.go:122] "RemoveContainer" containerID="058dc70cda927d69320615fa1aa05c81cef40094e73d7f970dd55bbb6ded88d0" Oct 30 17:30:19.962728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:19.962631 4622 scope.go:122] "RemoveContainer" containerID="d6f709fcc14f4c6cccf022c3f6125a7c41689d23d3ae6aa02c6fb943692c3868" Oct 30 17:30:19.962964 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:19.962941 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj_odh-ai-gateway-infra(53e32b91-04b3-40ca-97fe-1c3d1034659c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" Oct 30 17:30:20.050378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.050280 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:20.129141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.129109 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:20.166372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.166346 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:20.183039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.183004 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\" (UniqueName: \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\") pod \"158a45e1-a395-4f6c-9c76-12c4c71896f3\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " Oct 30 17:30:20.183217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.183170 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\") pod \"82f11800-3201-4095-bbc9-b3df311a6f21\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " Oct 30 17:30:20.183357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.183224 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\" (UniqueName: \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\") pod \"82f11800-3201-4095-bbc9-b3df311a6f21\" (UID: \"82f11800-3201-4095-bbc9-b3df311a6f21\") " Oct 30 17:30:20.184208 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.184178 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca" pod "82f11800-3201-4095-bbc9-b3df311a6f21" (UID: "82f11800-3201-4095-bbc9-b3df311a6f21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.190332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.187375 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn" pod "82f11800-3201-4095-bbc9-b3df311a6f21" (UID: "82f11800-3201-4095-bbc9-b3df311a6f21"). InnerVolumeSpecName "kube-api-access-f7gdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.190332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.188505 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5" pod "158a45e1-a395-4f6c-9c76-12c4c71896f3" (UID: "158a45e1-a395-4f6c-9c76-12c4c71896f3"). InnerVolumeSpecName "kube-api-access-gkmn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.284880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.284852 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\") pod \"158a45e1-a395-4f6c-9c76-12c4c71896f3\" (UID: \"158a45e1-a395-4f6c-9c76-12c4c71896f3\") " Oct 30 17:30:20.285322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.285255 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca" pod "158a45e1-a395-4f6c-9c76-12c4c71896f3" (UID: "158a45e1-a395-4f6c-9c76-12c4c71896f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.285442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.285372 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82f11800-3201-4095-bbc9-b3df311a6f21-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.285442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.285386 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7gdn\" (UniqueName: \"kubernetes.io/projected/82f11800-3201-4095-bbc9-b3df311a6f21-kube-api-access-f7gdn\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.285442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.285396 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/158a45e1-a395-4f6c-9c76-12c4c71896f3-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.285442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.285404 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmn5\" (UniqueName: \"kubernetes.io/projected/158a45e1-a395-4f6c-9c76-12c4c71896f3-kube-api-access-gkmn5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.974797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.974769 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" Oct 30 17:30:20.975242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.974810 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j" event={"ID":"82f11800-3201-4095-bbc9-b3df311a6f21","Type":"ContainerDied","Data":"352dd06c4292adc0e316e3491a5a534dd9b77e456d7a17fbc0da097d1167d1e1"} Oct 30 17:30:20.975242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.974873 4622 scope.go:122] "RemoveContainer" containerID="79506911e7b61e4099fc0157a31dc212f2ae19a5cbc7eb633aaa400275dbface" Oct 30 17:30:20.976480 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.976441 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" Oct 30 17:30:20.976553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.976477 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp" event={"ID":"158a45e1-a395-4f6c-9c76-12c4c71896f3","Type":"ContainerDied","Data":"280f00d067dd869f5338044181a5f79288271baf36f36f8b94010d893d48360c"} Oct 30 17:30:20.986175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:20.986158 4622 scope.go:122] "RemoveContainer" containerID="5913eb93ca7c1c995873d1e46641021167fb1feb41980cc84fb88c710000882b" Oct 30 17:30:21.030740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.030714 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:21.046531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.046502 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889690-r9k9j"] Oct 30 17:30:21.070551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.070527 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:21.080471 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.080443 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-306aed-29889690-rq8qp"] Oct 30 17:30:21.129110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.129085 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:21.194225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.194198 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\") pod \"53e32b91-04b3-40ca-97fe-1c3d1034659c\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " Oct 30 17:30:21.194410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.194327 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\" (UniqueName: \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\") pod \"53e32b91-04b3-40ca-97fe-1c3d1034659c\" (UID: \"53e32b91-04b3-40ca-97fe-1c3d1034659c\") " Oct 30 17:30:21.195130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.195103 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca" pod "53e32b91-04b3-40ca-97fe-1c3d1034659c" (UID: "53e32b91-04b3-40ca-97fe-1c3d1034659c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:21.197026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.196995 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq" pod "53e32b91-04b3-40ca-97fe-1c3d1034659c" (UID: "53e32b91-04b3-40ca-97fe-1c3d1034659c"). InnerVolumeSpecName "kube-api-access-gbxqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:21.296098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.296065 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e32b91-04b3-40ca-97fe-1c3d1034659c-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.296242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.296103 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxqq\" (UniqueName: \"kubernetes.io/projected/53e32b91-04b3-40ca-97fe-1c3d1034659c-kube-api-access-gbxqq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.988537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.988501 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" event={"ID":"53e32b91-04b3-40ca-97fe-1c3d1034659c","Type":"ContainerDied","Data":"a4cad26a1a6750cb34af1613c5cc4fc7f0a8a13ed1140eed278cf23147ddf2e3"} Oct 30 17:30:21.988992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.988540 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj" Oct 30 17:30:21.988992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:21.988572 4622 scope.go:122] "RemoveContainer" containerID="d6f709fcc14f4c6cccf022c3f6125a7c41689d23d3ae6aa02c6fb943692c3868" Oct 30 17:30:22.041519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:22.041494 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:22.053614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:22.053590 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8d946d-29889690-jj9qj"] Oct 30 17:30:22.373808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:22.373776 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" path="/var/lib/kubelet/pods/158a45e1-a395-4f6c-9c76-12c4c71896f3/volumes" Oct 30 17:30:22.374194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:22.374182 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" path="/var/lib/kubelet/pods/53e32b91-04b3-40ca-97fe-1c3d1034659c/volumes" Oct 30 17:30:22.374588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:22.374575 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" path="/var/lib/kubelet/pods/82f11800-3201-4095-bbc9-b3df311a6f21/volumes" Oct 30 17:30:24.534128 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:24.534095 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:30:37.983464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.983430 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:30:37.984205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984183 4622 cpu_manager.go:414] "containerMap: removing container" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984205 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984218 4622 cpu_manager.go:414] "containerMap: removing container" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984225 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984239 4622 cpu_manager.go:414] "containerMap: removing container" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984246 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984257 4622 cpu_manager.go:414] "containerMap: removing container" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984280 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984294 4622 cpu_manager.go:414] "containerMap: removing container" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984300 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984311 4622 cpu_manager.go:414] "containerMap: removing container" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984318 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984336 4622 cpu_manager.go:414] "containerMap: removing container" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984343 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984351 4622 cpu_manager.go:414] "containerMap: removing container" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984358 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984393 4622 cpu_manager.go:414] "containerMap: removing container" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.984422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984401 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984544 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984558 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984566 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984577 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984585 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984594 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e32b91-04b3-40ca-97fe-1c3d1034659c" containerName="cleanup" Oct 30 17:30:37.985159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.984604 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:37.988403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:37.988382 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.020307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.020254 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:30:38.024504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.024478 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:30:38.025230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.025210 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="158a45e1-a395-4f6c-9c76-12c4c71896f3" containerName="cleanup" Oct 30 17:30:38.025230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.025231 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f11800-3201-4095-bbc9-b3df311a6f21" containerName="cleanup" Oct 30 17:30:38.028545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.028528 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.045020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.044998 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:30:38.083723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.083692 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:30:38.088477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.088455 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.098872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.098841 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.098910 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.098956 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.098981 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpqr\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.099018 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.099039 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.099305 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.099379 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.099472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.099455 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.112655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.112629 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:30:38.200922 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.200885 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.200925 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.201093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201044 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.201093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201080 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7f6\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.201317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201150 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.201317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201210 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201230 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpqr\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201276 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.201317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201313 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201339 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.201568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201391 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201430 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201474 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.201568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201502 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.201829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201625 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201658 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.201829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201699 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.201829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201730 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.202002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201840 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.202002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201871 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.202002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201880 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.202002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.201938 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.202135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202003 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.202135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202055 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.202219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202135 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.202219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202169 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.202336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202218 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.202336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202291 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzmx\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.202404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202337 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.202435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202406 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.202482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202438 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.202792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.202772 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.203720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.203704 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.204016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.204000 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.224856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.224823 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.224971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.224894 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpqr\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.303467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303441 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:38.303818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303800 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.303875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303834 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.303911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303870 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.303911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303896 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.303988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.303966 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304038 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304115 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304159 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304184 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304200 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304215 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304251 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzmx\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304299 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304342 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304391 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304409 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304427 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304452 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7f6\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304497 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304532 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304786 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.304946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304900 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.304946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304900 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.305052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.304946 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.305109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.305082 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.305533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.305509 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.305684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.305654 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.305795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.305765 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.306762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.306743 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.306902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.306881 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.307138 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.307119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.307484 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.307466 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.327751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.327705 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.328049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.328024 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzmx\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.328323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.328304 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7f6\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.328393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.328346 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\") pod \"e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.351325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.351298 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:38.407475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.407442 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:38.617639 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:38.617607 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8523b8_4fec_423e_902b_9f01c2788e1d.slice/crio-ac693d7d11ddb412704200ce1cdf26854f8a81717f331984ad136f17b2f886b5 WatchSource:0}: Error finding container ac693d7d11ddb412704200ce1cdf26854f8a81717f331984ad136f17b2f886b5: Status 404 returned error can't find the container with id ac693d7d11ddb412704200ce1cdf26854f8a81717f331984ad136f17b2f886b5 Oct 30 17:30:38.619278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.619241 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:30:38.620216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.620186 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.620305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.620246 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.620358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.620312 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.672435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.672409 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:30:38.677520 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:38.677496 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66f2487_4ccf_4817_9646_b23ac17e39e4.slice/crio-510c1e756f4a5d5c8fcaed3f9e4a4983b7f446df04f60afddd4f9158d4dc5a0e WatchSource:0}: Error finding container 510c1e756f4a5d5c8fcaed3f9e4a4983b7f446df04f60afddd4f9158d4dc5a0e: Status 404 returned error can't find the container with id 510c1e756f4a5d5c8fcaed3f9e4a4983b7f446df04f60afddd4f9158d4dc5a0e Oct 30 17:30:38.680286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.680228 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.680368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.680313 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.680368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.680346 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.727995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.727955 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:30:38.733705 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:38.733654 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a9d4d4_0e3e_4524_98a2_e62ac8265d5f.slice/crio-f1ba53d2ebc7f5ee0eb78719ade51ed41c9266618133a3ba9011183874f9b010 WatchSource:0}: Error finding container f1ba53d2ebc7f5ee0eb78719ade51ed41c9266618133a3ba9011183874f9b010: Status 404 returned error can't find the container with id f1ba53d2ebc7f5ee0eb78719ade51ed41c9266618133a3ba9011183874f9b010 Oct 30 17:30:38.737028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.736996 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.737123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.737074 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:38.737123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:38.737118 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:39.109351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.109314 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" event={"ID":"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f","Type":"ContainerStarted","Data":"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be"} Oct 30 17:30:39.109779 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.109381 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" event={"ID":"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f","Type":"ContainerStarted","Data":"f1ba53d2ebc7f5ee0eb78719ade51ed41c9266618133a3ba9011183874f9b010"} Oct 30 17:30:39.112811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.112786 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" event={"ID":"f66f2487-4ccf-4817-9646-b23ac17e39e4","Type":"ContainerStarted","Data":"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a"} Oct 30 17:30:39.113015 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.112995 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" event={"ID":"f66f2487-4ccf-4817-9646-b23ac17e39e4","Type":"ContainerStarted","Data":"510c1e756f4a5d5c8fcaed3f9e4a4983b7f446df04f60afddd4f9158d4dc5a0e"} Oct 30 17:30:39.115292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.115253 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" event={"ID":"df8523b8-4fec-423e-902b-9f01c2788e1d","Type":"ContainerStarted","Data":"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435"} Oct 30 17:30:39.115450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.115432 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" event={"ID":"df8523b8-4fec-423e-902b-9f01c2788e1d","Type":"ContainerStarted","Data":"ac693d7d11ddb412704200ce1cdf26854f8a81717f331984ad136f17b2f886b5"} Oct 30 17:30:39.187927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.187863 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podStartSLOduration=1.187840954 podStartE2EDuration="1.187840954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.16058743 +0000 UTC m=+1066.938059385" watchObservedRunningTime="2026-10-30 17:30:39.187840954 +0000 UTC m=+1066.965312911" Oct 30 17:30:39.214115 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.214022 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podStartSLOduration=2.213997102 podStartE2EDuration="2.213997102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.209893512 +0000 UTC m=+1066.987365507" watchObservedRunningTime="2026-10-30 17:30:39.213997102 +0000 UTC m=+1066.991469050" Oct 30 17:30:39.266918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.266857 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podStartSLOduration=2.26683621 podStartE2EDuration="2.26683621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:39.265761496 +0000 UTC m=+1067.043233452" watchObservedRunningTime="2026-10-30 17:30:39.26683621 +0000 UTC m=+1067.044308167" Oct 30 17:30:39.310289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.306378 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:39.310289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.306553 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.310289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.306600 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:39.354655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.354619 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:39.358439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.358404 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.358656 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.358629 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:39.408526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.408411 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:39.411168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.411134 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:39.411448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:39.411426 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:40.306686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.306649 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.307280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.307234 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:40.354728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.354689 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.355027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.354990 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:40.409795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.409360 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:40.409795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.409419 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:40.868079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.868039 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 17:30:40.916283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.904453 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:40.933655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.933623 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0e7403\"/\"distinct-2-w4-0e7403-kserve-self-signed-certs\"" Oct 30 17:30:40.933862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.933844 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 17:30:41.059309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:40.972522 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkfw\" (UniqueName: \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.072995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.072962 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.073345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.073321 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.073522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.073504 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.073656 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.073641 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.073837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.073821 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.176647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.176544 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkfw\" (UniqueName: \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.213391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.213359 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.213733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.213708 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.213945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.213929 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.232533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.214467 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.233137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.215119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.233300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.232394 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.233434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.232770 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.233647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.233631 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.234221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.234054 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.244302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.244237 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.255350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.255277 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkfw\" (UniqueName: \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\") pod \"distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:41.306064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.305636 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.306064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.305694 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:41.353664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.353630 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.354251 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.354213 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:41.410080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.409634 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:41.410080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.409694 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:41.429009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:41.428699 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:42.224284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.220451 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 17:30:42.232434 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:42.232384 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb8de23_92b8_4bc0_8c35_16e51709a6a2.slice/crio-01135e2fff25e7997779c8a03e52e20b2dba3630505fe6baea4221d4c23f93cc WatchSource:0}: Error finding container 01135e2fff25e7997779c8a03e52e20b2dba3630505fe6baea4221d4c23f93cc: Status 404 returned error can't find the container with id 01135e2fff25e7997779c8a03e52e20b2dba3630505fe6baea4221d4c23f93cc Oct 30 17:30:42.313824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.313784 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.314121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.314079 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:42.354418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.354359 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.355023 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.354992 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:42.414334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.411725 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:42.414334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:42.411782 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:43.304973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.304935 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.305194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.305003 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:43.307293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.307245 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" event={"ID":"2cb8de23-92b8-4bc0-8c35-16e51709a6a2","Type":"ContainerStarted","Data":"782f385df7c343f199284c11237e5215e3aa882c883cb40b23494c633c8ee4b6"} Oct 30 17:30:43.307493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.307473 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" event={"ID":"2cb8de23-92b8-4bc0-8c35-16e51709a6a2","Type":"ContainerStarted","Data":"01135e2fff25e7997779c8a03e52e20b2dba3630505fe6baea4221d4c23f93cc"} Oct 30 17:30:43.398287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.375958 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podStartSLOduration=3.375938244 podStartE2EDuration="3.375938244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:43.374421448 +0000 UTC m=+1071.151893406" watchObservedRunningTime="2026-10-30 17:30:43.375938244 +0000 UTC m=+1071.153410201" Oct 30 17:30:43.421314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.401007 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.421582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.421554 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:43.422296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.412711 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:43.422500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.422476 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:43.430406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:43.430372 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:44.304763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.304725 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.305056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.305027 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:44.359979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.359942 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.360288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.360233 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:44.410732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.410697 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:44.411348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:44.411320 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:45.304453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.304418 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.304671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.304477 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 17:30:45.351987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.351954 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.352296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.352237 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 17:30:45.410292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.410226 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 17:30:45.410566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:45.410538 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 17:30:46.508998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.508965 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:46.510203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.509978 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:46.512585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.512565 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:30:46.916286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.915581 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:46.920284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.917041 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:46.920284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:46.918447 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:30:47.049558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:47.049526 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:47.051333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:47.051308 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:47.052867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:47.052849 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:30:51.430430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:51.430400 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:51.442927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:51.442901 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:52.417175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.417144 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:30:52.842140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.842108 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:30:52.847856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.847836 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.880159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.880128 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:30:52.948408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948378 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948418 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bx8\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948446 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948566 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948630 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948671 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948807 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948789 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948836 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:52.948918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:52.948905 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050516 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050550 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050586 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050626 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050653 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bx8\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050685 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.050784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050748 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050822 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050873 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.050945 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.051134 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.051171 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.051206 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.051611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.051591 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.053254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.053225 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.053421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.053403 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.074157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.074136 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.074328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.074307 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bx8\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\") pod \"e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.165802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.165523 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:53.484724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.484697 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:30:53.484860 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:30:53.484799 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f21d5fe_e150_4cd3_b648_25a0870d4bf9.slice/crio-8c8da560c330a98155c5a0d6724ca351625945478ca8eee655fd006c4f019208 WatchSource:0}: Error finding container 8c8da560c330a98155c5a0d6724ca351625945478ca8eee655fd006c4f019208: Status 404 returned error can't find the container with id 8c8da560c330a98155c5a0d6724ca351625945478ca8eee655fd006c4f019208 Oct 30 17:30:53.487138 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.487112 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:53.487214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.487169 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:53.487214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:53.487202 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:30:54.420220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:54.420185 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" event={"ID":"7f21d5fe-e150-4cd3-b648-25a0870d4bf9","Type":"ContainerStarted","Data":"3d8363a7a37873370212616a937234f9182a4cf1d7f39da4172146507da51e87"} Oct 30 17:30:54.420744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:54.420231 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" event={"ID":"7f21d5fe-e150-4cd3-b648-25a0870d4bf9","Type":"ContainerStarted","Data":"8c8da560c330a98155c5a0d6724ca351625945478ca8eee655fd006c4f019208"} Oct 30 17:30:54.496532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:54.496482 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podStartSLOduration=2.496469097 podStartE2EDuration="2.496469097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:54.481160849 +0000 UTC m=+1082.258632804" watchObservedRunningTime="2026-10-30 17:30:54.496469097 +0000 UTC m=+1082.273941051" Oct 30 17:30:54.533770 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:30:54.533734 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:30:55.166475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:55.166431 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:30:55.168649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:55.168616 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:55.168812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:55.168698 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:30:56.167111 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:56.167071 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:56.167767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:56.167712 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:30:57.166728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:57.166688 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:57.166936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:57.166751 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:30:58.171722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:58.171686 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:58.172243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:58.171750 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:30:59.166071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:59.166032 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:30:59.166257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:30:59.166094 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:31:00.166720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:00.166678 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 17:31:00.167102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:00.166746 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 17:31:01.170536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:01.170508 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:31:01.170946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:01.170842 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:31:01.171408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:01.171391 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:31:05.447857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:05.447809 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.619409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:05.619354 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.781649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:05.781500 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.859513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:05.858389 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:05.925447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:05.924381 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:06.228291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.202893 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:31:06.474287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.453479 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:06.599130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.599098 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:31:06.636535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.636479 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:06.642819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.642790 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:31:06.724329 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.724145 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:31:06.728386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:06.728344 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:31:07.904727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.903764 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:31:07.920589 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.920556 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.930335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.930305 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:31:07.934929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.934902 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:31:07.939066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.939041 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.973989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.973935 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974062 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974116 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974170 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974239 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974279 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974314 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974337 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26v8\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974379 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974431 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.974774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974463 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9xt\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.974774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.974484 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.975400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.975374 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.975560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.975410 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.975560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.975444 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.975560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.975496 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:07.975560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.975539 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:07.977803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:07.977778 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:31:08.077008 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.076977 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077029 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077089 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077130 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077165 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077188 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077224 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077250 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26v8\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077315 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077375 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077410 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9xt\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077434 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077492 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077515 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077550 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077603 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077622 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077639 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077650 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077674 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.077772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077712 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.078332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.077842 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.078332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.078199 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.078332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.078318 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.078691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.078646 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.078691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.078674 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.078913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.078745 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.079155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.079128 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.080992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.080968 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.081094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.081043 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.082137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.082100 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.082237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.082132 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.102560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.102510 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26v8\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.103467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.103443 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.103928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.103902 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9xt\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.104730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.104709 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\") pod \"e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.241030 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.240964 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:08.255669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.255647 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:08.577978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.577653 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:31:08.580168 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:31:08.580098 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ca6351_9809_4dde_8040_fe18ad9e41fa.slice/crio-daec2ea5ec22b5327411b46b0b16d9093b9dc7fed28f8841a6ccb06a5c58e16b WatchSource:0}: Error finding container daec2ea5ec22b5327411b46b0b16d9093b9dc7fed28f8841a6ccb06a5c58e16b: Status 404 returned error can't find the container with id daec2ea5ec22b5327411b46b0b16d9093b9dc7fed28f8841a6ccb06a5c58e16b Oct 30 17:31:08.583410 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:31:08.582036 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623ee1b6_3540_4022_8b7f_ab0696b1423b.slice/crio-8c98cd0547a19aced8a9a5ff7a775149cd38be612a789b8dad67f9f23766c8f5 WatchSource:0}: Error finding container 8c98cd0547a19aced8a9a5ff7a775149cd38be612a789b8dad67f9f23766c8f5: Status 404 returned error can't find the container with id 8c98cd0547a19aced8a9a5ff7a775149cd38be612a789b8dad67f9f23766c8f5 Oct 30 17:31:08.584469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.584215 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.584469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.584357 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.584469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.584406 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.585456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.585391 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:31:08.588704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.587778 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.588704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.587845 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.588704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.587885 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:08.616814 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.616791 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" event={"ID":"f7ca6351-9809-4dde-8040-fe18ad9e41fa","Type":"ContainerStarted","Data":"daec2ea5ec22b5327411b46b0b16d9093b9dc7fed28f8841a6ccb06a5c58e16b"} Oct 30 17:31:08.618188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:08.618166 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" event={"ID":"623ee1b6-3540-4022-8b7f-ab0696b1423b","Type":"ContainerStarted","Data":"8c98cd0547a19aced8a9a5ff7a775149cd38be612a789b8dad67f9f23766c8f5"} Oct 30 17:31:09.634315 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:09.633390 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" event={"ID":"623ee1b6-3540-4022-8b7f-ab0696b1423b","Type":"ContainerStarted","Data":"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568"} Oct 30 17:31:09.637191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:09.637142 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" event={"ID":"f7ca6351-9809-4dde-8040-fe18ad9e41fa","Type":"ContainerStarted","Data":"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83"} Oct 30 17:31:09.712414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:09.712367 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podStartSLOduration=2.712353157 podStartE2EDuration="2.712353157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:09.71081711 +0000 UTC m=+1097.488289066" watchObservedRunningTime="2026-10-30 17:31:09.712353157 +0000 UTC m=+1097.489825111" Oct 30 17:31:10.246287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.244624 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:10.257682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.257651 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:10.263474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.263438 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:10.263758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.263729 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:10.280452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.280416 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:10.280722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:10.280694 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:11.244662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:11.244623 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:11.245661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:11.245241 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:11.259658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:11.259623 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:11.260050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:11.260022 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:12.248287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:12.244656 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:12.248287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:12.244716 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:12.275712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:12.275678 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:12.276482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:12.276450 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:13.248669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:13.248631 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:13.249295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:13.249244 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:13.258620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:13.258592 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:13.258820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:13.258797 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:14.243296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:14.242749 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:14.243296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:14.242812 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:14.268403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:14.268362 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:14.268978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:14.268948 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:15.241899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:15.241855 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 17:31:15.242066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:15.241931 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 17:31:15.256426 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:15.256393 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:31:15.256542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:15.256450 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 17:31:17.251307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.250400 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:17.251307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.250494 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.304906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.304871 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:17.305210 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.305177 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.381582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.381550 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:17.383248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.383222 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:17.390462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.390442 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:31:17.685287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.681109 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podStartSLOduration=10.681084393999999 podStartE2EDuration="10.681084394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:09.788883308 +0000 UTC m=+1097.566355294" watchObservedRunningTime="2026-10-30 17:31:17.681084394 +0000 UTC m=+1105.458556351" Oct 30 17:31:17.690079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.690050 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:17.773307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.770488 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.773307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.771655 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.787615 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.787576 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:17.968339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.966673 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:17.968339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.966818 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:31:17.975372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.975340 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:17.975597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:17.975569 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:18.115403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:18.115356 4622 patch_prober.go:29] interesting pod/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:31:18.115614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:18.115431 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:23.227543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.227460 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:31:23.233166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.233144 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.274078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.274041 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:31:23.334821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.334784 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.334992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.334854 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.334992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.334913 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.334992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.334939 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfhl\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.334992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.334972 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.335142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.335091 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.335187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.335151 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.335187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.335181 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.335350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.335327 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436379 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436455 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436477 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436504 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfhl\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436640 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436737 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436810 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436864 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436947 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.436990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.436949 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.437090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.437030 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.437283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.437239 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.437283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.437256 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.437481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.437383 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.438983 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.438953 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.439103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.439087 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.462106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.462079 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.462195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.462119 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfhl\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\") pod \"e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.548745 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.548720 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:23.866322 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:31:23.866275 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0c7c85_8afb_483e_aaba_b37da00849c5.slice/crio-f844e1521b1eee8b778225b89f934ea017b625367e1fa417579f3665e8a076cf WatchSource:0}: Error finding container f844e1521b1eee8b778225b89f934ea017b625367e1fa417579f3665e8a076cf: Status 404 returned error can't find the container with id f844e1521b1eee8b778225b89f934ea017b625367e1fa417579f3665e8a076cf Oct 30 17:31:23.866991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.866967 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:31:23.868779 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.868751 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:23.868859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.868811 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:23.868859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.868836 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:31:23.921335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:23.921311 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" event={"ID":"2c0c7c85-8afb-483e-aaba-b37da00849c5","Type":"ContainerStarted","Data":"f844e1521b1eee8b778225b89f934ea017b625367e1fa417579f3665e8a076cf"} Oct 30 17:31:24.534085 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:31:24.534050 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:31:24.930691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:24.930614 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" event={"ID":"2c0c7c85-8afb-483e-aaba-b37da00849c5","Type":"ContainerStarted","Data":"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec"} Oct 30 17:31:25.006031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:25.005980 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podStartSLOduration=2.005966654 podStartE2EDuration="2.005966654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:25.003565819 +0000 UTC m=+1112.781037775" watchObservedRunningTime="2026-10-30 17:31:25.005966654 +0000 UTC m=+1112.783438609" Oct 30 17:31:25.565636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:25.565594 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:25.567092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:25.567066 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:25.567207 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:25.567120 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:26.564285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:26.563667 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:26.564285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:26.563730 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:27.560293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:27.554674 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:27.560293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:27.554734 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:28.555355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:28.555304 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:28.565479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:28.565366 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:29.563673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:29.563635 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:29.564681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:29.564642 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:30.549050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:30.549013 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:31:30.549447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:30.549063 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 17:31:31.554236 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:31.554206 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:31.554699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:31.554535 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:31.555201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:31.555187 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:31:33.743399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.743346 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.785537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.785493 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.831960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.831373 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.857391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.857346 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:33.865425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.865337 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 17:31:33.918388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.918347 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 17:31:33.918936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:33.918648 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.076376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.076316 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.204558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.204339 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.747531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.747491 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.764308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.748427 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:31:34.786497 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.786458 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.807783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.807754 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:31:34.893295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.892826 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 17:31:34.893295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.893251 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:31:34.963285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.954799 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.963285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.955248 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:31:34.963285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.961082 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:34.963285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:34.962141 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:31:35.751287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:35.750345 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.831401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:35.831357 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:31:35.895644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:35.895600 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:35.963526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:35.963482 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:31:35.989563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:35.982049 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.081395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.081352 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:31:36.171835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.171804 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:31:36.347409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.340339 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:31:36.428454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.428422 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:31:36.431946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.431924 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:31:36.448885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:31:36.447183 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:31:54.644904 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:31:54.644859 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:32:10.164635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.164600 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc"] Oct 30 17:32:10.168600 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.168578 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.187598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.187570 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc"] Oct 30 17:32:10.195764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.195729 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:32:10.195764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.195738 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:32:10.294343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.294306 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.294510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.294353 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.294510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.294450 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkxfl\" (UniqueName: \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.396124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.396081 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.396339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.396136 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.396339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.396190 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkxfl\" (UniqueName: \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.396862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.396840 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.398501 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.398476 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.419712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.419645 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkxfl\" (UniqueName: \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\") pod \"maas-api-revoke-sub-da079b02-wqlgc\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.482306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.482276 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:10.787137 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:32:10.787066 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f69d903_2ca0_48bb_8065_b3108afba0e2.slice/crio-f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3 WatchSource:0}: Error finding container f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3: Status 404 returned error can't find the container with id f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3 Oct 30 17:32:10.787384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:10.787359 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc"] Oct 30 17:32:11.642331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:11.641853 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" event={"ID":"1f69d903-2ca0-48bb-8065-b3108afba0e2","Type":"ContainerStarted","Data":"4c7e6bf02651d4460e0ed945cdd6071d889f0905f8cf47f16c5c11b8a21b7144"} Oct 30 17:32:11.642331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:11.641912 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" event={"ID":"1f69d903-2ca0-48bb-8065-b3108afba0e2","Type":"ContainerStarted","Data":"f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3"} Oct 30 17:32:11.684088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:11.683996 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" podStartSLOduration=1.6839811390000001 podStartE2EDuration="1.683981139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:11.681646638 +0000 UTC m=+1159.459118594" watchObservedRunningTime="2026-10-30 17:32:11.683981139 +0000 UTC m=+1159.461453123" Oct 30 17:32:12.606839 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.606806 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb"] Oct 30 17:32:12.611162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.611139 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.628851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.628826 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb"] Oct 30 17:32:12.652235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.652204 4622 generic.go:356] "Generic (PLEG): container finished" podID="1f69d903-2ca0-48bb-8065-b3108afba0e2" containerID="4c7e6bf02651d4460e0ed945cdd6071d889f0905f8cf47f16c5c11b8a21b7144" exitCode=0 Oct 30 17:32:12.652728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.652301 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" event={"ID":"1f69d903-2ca0-48bb-8065-b3108afba0e2","Type":"ContainerDied","Data":"4c7e6bf02651d4460e0ed945cdd6071d889f0905f8cf47f16c5c11b8a21b7144"} Oct 30 17:32:12.728239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.728206 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.728425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.728345 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.728425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.728398 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m67q\" (UniqueName: \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.829198 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.829164 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.829381 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.829280 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m67q\" (UniqueName: \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.829506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.829484 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.829994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.829971 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.832480 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.832447 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.862056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.861995 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m67q\" (UniqueName: \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\") pod \"maas-api-revoke-sub-ef32ce26-vnhpb\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:12.925453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:12.925429 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:13.227673 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:32:13.227639 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3718ac1_af59_4f5e_9945_6127e0f7abea.slice/crio-6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118 WatchSource:0}: Error finding container 6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118: Status 404 returned error can't find the container with id 6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118 Oct 30 17:32:13.228884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.228853 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb"] Oct 30 17:32:13.662382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.662348 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" event={"ID":"a3718ac1-af59-4f5e-9945-6127e0f7abea","Type":"ContainerStarted","Data":"b37c03fe132f7940ed507caa6df5cf0d3e6332e94fd05cadd30465fcd665c344"} Oct 30 17:32:13.662805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.662409 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" event={"ID":"a3718ac1-af59-4f5e-9945-6127e0f7abea","Type":"ContainerStarted","Data":"6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118"} Oct 30 17:32:13.704833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.704785 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" podStartSLOduration=1.704772016 podStartE2EDuration="1.704772016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:13.702088798 +0000 UTC m=+1161.479560754" watchObservedRunningTime="2026-10-30 17:32:13.704772016 +0000 UTC m=+1161.482243986" Oct 30 17:32:13.797303 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.797275 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:13.943534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.943504 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\" (UniqueName: \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\") pod \"1f69d903-2ca0-48bb-8065-b3108afba0e2\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " Oct 30 17:32:13.943696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.943602 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\") pod \"1f69d903-2ca0-48bb-8065-b3108afba0e2\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " Oct 30 17:32:13.943778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.943702 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\") pod \"1f69d903-2ca0-48bb-8065-b3108afba0e2\" (UID: \"1f69d903-2ca0-48bb-8065-b3108afba0e2\") " Oct 30 17:32:13.943969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.943940 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca" pod "1f69d903-2ca0-48bb-8065-b3108afba0e2" (UID: "1f69d903-2ca0-48bb-8065-b3108afba0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:13.944085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.944057 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:13.945673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.945647 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca" pod "1f69d903-2ca0-48bb-8065-b3108afba0e2" (UID: "1f69d903-2ca0-48bb-8065-b3108afba0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:13.945760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:13.945678 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl" pod "1f69d903-2ca0-48bb-8065-b3108afba0e2" (UID: "1f69d903-2ca0-48bb-8065-b3108afba0e2"). InnerVolumeSpecName "kube-api-access-dkxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:14.045308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.045281 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkxfl\" (UniqueName: \"kubernetes.io/projected/1f69d903-2ca0-48bb-8065-b3108afba0e2-kube-api-access-dkxfl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:14.045308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.045305 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69d903-2ca0-48bb-8065-b3108afba0e2-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:14.670220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.670192 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" Oct 30 17:32:14.670220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.670201 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc" event={"ID":"1f69d903-2ca0-48bb-8065-b3108afba0e2","Type":"ContainerDied","Data":"f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3"} Oct 30 17:32:14.670665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.670243 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c5ab1158aef71d7baca0d24abba1f424dea909ff6cb2104795f6e1c19ed7f3" Oct 30 17:32:14.672009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.671983 4622 generic.go:356] "Generic (PLEG): container finished" podID="a3718ac1-af59-4f5e-9945-6127e0f7abea" containerID="b37c03fe132f7940ed507caa6df5cf0d3e6332e94fd05cadd30465fcd665c344" exitCode=0 Oct 30 17:32:14.672144 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:14.672055 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" event={"ID":"a3718ac1-af59-4f5e-9945-6127e0f7abea","Type":"ContainerDied","Data":"b37c03fe132f7940ed507caa6df5cf0d3e6332e94fd05cadd30465fcd665c344"} Oct 30 17:32:15.863081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.863058 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:15.966973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.966892 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\" (UniqueName: \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\") pod \"a3718ac1-af59-4f5e-9945-6127e0f7abea\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " Oct 30 17:32:15.967117 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.966990 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\") pod \"a3718ac1-af59-4f5e-9945-6127e0f7abea\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " Oct 30 17:32:15.967177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.967157 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\") pod \"a3718ac1-af59-4f5e-9945-6127e0f7abea\" (UID: \"a3718ac1-af59-4f5e-9945-6127e0f7abea\") " Oct 30 17:32:15.967517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.967487 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca" pod "a3718ac1-af59-4f5e-9945-6127e0f7abea" (UID: "a3718ac1-af59-4f5e-9945-6127e0f7abea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:15.967776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.967759 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:15.969047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.969014 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca" pod "a3718ac1-af59-4f5e-9945-6127e0f7abea" (UID: "a3718ac1-af59-4f5e-9945-6127e0f7abea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:15.969047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:15.969041 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q" pod "a3718ac1-af59-4f5e-9945-6127e0f7abea" (UID: "a3718ac1-af59-4f5e-9945-6127e0f7abea"). InnerVolumeSpecName "kube-api-access-7m67q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:16.068941 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:16.068914 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3718ac1-af59-4f5e-9945-6127e0f7abea-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:16.068941 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:16.068938 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m67q\" (UniqueName: \"kubernetes.io/projected/a3718ac1-af59-4f5e-9945-6127e0f7abea-kube-api-access-7m67q\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:32:16.686741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:16.686662 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" Oct 30 17:32:16.686741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:16.686681 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb" event={"ID":"a3718ac1-af59-4f5e-9945-6127e0f7abea","Type":"ContainerDied","Data":"6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118"} Oct 30 17:32:16.686741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:16.686727 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b275112962cff298ed28e905c2cc52757ed6a751d828f4f31d2dd77cc7db118" Oct 30 17:32:24.534712 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:32:24.534672 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:32:26.225067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.225039 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 17:32:26.225845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.225824 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a3718ac1-af59-4f5e-9945-6127e0f7abea" containerName="revoke-keys" Oct 30 17:32:26.225937 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.225847 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3718ac1-af59-4f5e-9945-6127e0f7abea" containerName="revoke-keys" Oct 30 17:32:26.225937 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.225891 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1f69d903-2ca0-48bb-8065-b3108afba0e2" containerName="revoke-keys" Oct 30 17:32:26.225937 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.225899 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f69d903-2ca0-48bb-8065-b3108afba0e2" containerName="revoke-keys" Oct 30 17:32:26.226098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.226000 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3718ac1-af59-4f5e-9945-6127e0f7abea" containerName="revoke-keys" Oct 30 17:32:26.226098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.226010 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f69d903-2ca0-48bb-8065-b3108afba0e2" containerName="revoke-keys" Oct 30 17:32:26.229727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.229703 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.247311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.247284 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 17:32:26.254050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.254016 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"distinct-w2-506124-kserve-self-signed-certs\"" Oct 30 17:32:26.309096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.309058 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.309386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.309365 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.309555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.309538 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.309756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.309737 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbvc\" (UniqueName: \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.309899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.309886 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.310055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.310041 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.425225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.425188 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.426215 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.426001 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.426645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.426628 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.427923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.427066 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.428417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.428399 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.429381 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.429363 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.430052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.429866 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbvc\" (UniqueName: \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.430730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.430711 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.432304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.432285 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.461190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.461125 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.462451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.462427 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.465298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.463913 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbvc\" (UniqueName: \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\") pod \"distinct-w2-506124-kserve-54459649df-dcxs8\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.544923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.544861 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:26.854351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:26.854309 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 17:32:27.763224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:27.763159 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" event={"ID":"3427dc4e-2da1-48d2-bfe3-859ab0d20550","Type":"ContainerStarted","Data":"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4"} Oct 30 17:32:27.763667 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:27.763242 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" event={"ID":"3427dc4e-2da1-48d2-bfe3-859ab0d20550","Type":"ContainerStarted","Data":"f476b63c092f67953bfb471a5dfebc0aebb767d1193c01cd94d4f27034f2a190"} Oct 30 17:32:27.808427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:27.808377 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podStartSLOduration=1.808364531 podStartE2EDuration="1.808364531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:27.80576277 +0000 UTC m=+1175.583234724" watchObservedRunningTime="2026-10-30 17:32:27.808364531 +0000 UTC m=+1175.585836486" Oct 30 17:32:28.545221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:28.545191 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:31.846285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:31.840870 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:31.846285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:31.840955 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:31.947339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:31.947295 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:31.947646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:31.947597 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:31.991401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:31.991341 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:32:32.075393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.075346 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:32.110659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.110562 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:32.110904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.110880 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:32.192286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.191393 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:32.192286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.191455 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:32.220292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.219360 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:32:32.220292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:32.219475 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:36.545526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:36.545490 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:36.558308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:36.558248 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:36.884484 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:36.884413 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:48.879287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:48.878359 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:48.908287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:48.907363 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:48.999110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:48.999049 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:32:49.001487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:48.999409 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:32:49.082610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.081380 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:49.441288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.440372 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 17:32:49.879454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.879403 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:49.880804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.880761 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:49.911417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.911372 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:49.912135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:49.912111 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:32:50.003394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.003350 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.035899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.035867 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:32:50.105301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.097556 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.105301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.099125 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:32:50.165335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.144881 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 17:32:50.344428 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.344369 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.573346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.445316 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.632043 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.632008 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:32:50.898461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.881095 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:50.915573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:50.914354 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.070291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.067092 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:32:51.096518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.070601 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.248347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.223512 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:32:51.257921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.257889 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:32:51.361822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.361430 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.366782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.366757 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:32:51.367110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.367091 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:32:51.585850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.585802 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:51.699050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:51.699011 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:32:52.099587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.099372 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.100085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.099788 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:32:52.161192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.145486 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:32:52.161192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.146211 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:32:52.428402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.390577 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:32:52.857559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.857530 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:32:52.857823 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.857811 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:32:52.857932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.857923 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:32:52.858029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:52.858021 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:32:54.535674 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:32:54.535638 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:32:57.660190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.660155 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8848-b68665"] Oct 30 17:32:57.664342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.664325 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-b68665" Oct 30 17:32:57.682752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.682728 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-b68665"] Oct 30 17:32:57.690963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.690923 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5c6gd\"" Oct 30 17:32:57.699547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.699511 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7flq\" (UniqueName: \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\") pod \"test-curl-8848-b68665\" (UID: \"ccf363b6-8107-4ed0-87a9-09184173039e\") " pod="opendatahub/test-curl-8848-b68665" Oct 30 17:32:57.800970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.800934 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7flq\" (UniqueName: \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\") pod \"test-curl-8848-b68665\" (UID: \"ccf363b6-8107-4ed0-87a9-09184173039e\") " pod="opendatahub/test-curl-8848-b68665" Oct 30 17:32:57.824195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.824164 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7flq\" (UniqueName: \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\") pod \"test-curl-8848-b68665\" (UID: \"ccf363b6-8107-4ed0-87a9-09184173039e\") " pod="opendatahub/test-curl-8848-b68665" Oct 30 17:32:57.977427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:57.977351 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-b68665" Oct 30 17:32:58.275555 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:32:58.275519 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccf363b6_8107_4ed0_87a9_09184173039e.slice/crio-0c2d40187b32e5c163c32c157f0812a9ba1ba448218c9f4e8d91feb10abaddbc WatchSource:0}: Error finding container 0c2d40187b32e5c163c32c157f0812a9ba1ba448218c9f4e8d91feb10abaddbc: Status 404 returned error can't find the container with id 0c2d40187b32e5c163c32c157f0812a9ba1ba448218c9f4e8d91feb10abaddbc Oct 30 17:32:58.275753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:58.275731 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-b68665"] Oct 30 17:32:58.896603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:58.896570 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-b68665" event={"ID":"ccf363b6-8107-4ed0-87a9-09184173039e","Type":"ContainerStarted","Data":"2f2a573c650dc3af4805c90bf596ad6109402e56e2c8b75b491b9504840df73b"} Oct 30 17:32:58.896603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:58.896608 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-b68665" event={"ID":"ccf363b6-8107-4ed0-87a9-09184173039e","Type":"ContainerStarted","Data":"0c2d40187b32e5c163c32c157f0812a9ba1ba448218c9f4e8d91feb10abaddbc"} Oct 30 17:32:58.935580 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:58.935536 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8848-b68665" podStartSLOduration=1.464051834 podStartE2EDuration="1.935523143s" totalImagesPullingTime="471.471309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:58.934633987 +0000 UTC m=+1206.712105943" watchObservedRunningTime="2026-10-30 17:32:58.935523143 +0000 UTC m=+1206.712995097" Oct 30 17:32:59.841718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:59.841681 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8848-b68665"] Oct 30 17:32:59.847581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:32:59.847556 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8848-b68665"] Oct 30 17:33:00.010533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.010503 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8848-bbb454"] Oct 30 17:33:00.011159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.011143 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" containerName="test-curl-8848-b68665" Oct 30 17:33:00.011159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.011159 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" containerName="test-curl-8848-b68665" Oct 30 17:33:00.011291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.011257 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" containerName="test-curl-8848-b68665" Oct 30 17:33:00.014924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.014908 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:00.030690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.030664 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-bbb454"] Oct 30 17:33:00.129976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.129895 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrpd\" (UniqueName: \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\") pod \"test-curl-8848-bbb454\" (UID: \"ef624704-f043-4f4c-8ee1-d64e4f95551a\") " pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:00.231455 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.231422 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrpd\" (UniqueName: \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\") pod \"test-curl-8848-bbb454\" (UID: \"ef624704-f043-4f4c-8ee1-d64e4f95551a\") " pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:00.254642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.254607 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrpd\" (UniqueName: \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\") pod \"test-curl-8848-bbb454\" (UID: \"ef624704-f043-4f4c-8ee1-d64e4f95551a\") " pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:00.327916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.327885 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:00.644808 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:00.644774 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef624704_f043_4f4c_8ee1_d64e4f95551a.slice/crio-973a3bd49c7bbb8e27552d7ef8eba2f101cfc30f1c2be8351fb9a57b683ff267 WatchSource:0}: Error finding container 973a3bd49c7bbb8e27552d7ef8eba2f101cfc30f1c2be8351fb9a57b683ff267: Status 404 returned error can't find the container with id 973a3bd49c7bbb8e27552d7ef8eba2f101cfc30f1c2be8351fb9a57b683ff267 Oct 30 17:33:00.644957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.644868 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8848-bbb454"] Oct 30 17:33:00.911392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.911308 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-bbb454" event={"ID":"ef624704-f043-4f4c-8ee1-d64e4f95551a","Type":"ContainerStarted","Data":"fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05"} Oct 30 17:33:00.911392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.911356 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8848-bbb454" event={"ID":"ef624704-f043-4f4c-8ee1-d64e4f95551a","Type":"ContainerStarted","Data":"973a3bd49c7bbb8e27552d7ef8eba2f101cfc30f1c2be8351fb9a57b683ff267"} Oct 30 17:33:00.911591 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.911511 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8848-b68665" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" containerName="test-curl-8848-b68665" containerID="cri-o://2f2a573c650dc3af4805c90bf596ad6109402e56e2c8b75b491b9504840df73b" gracePeriod=2 Oct 30 17:33:00.950106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:00.950060 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8848-bbb454" podStartSLOduration=1.9500479419999999 podStartE2EDuration="1.950047942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:00.949058556 +0000 UTC m=+1208.726530510" watchObservedRunningTime="2026-10-30 17:33:00.950047942 +0000 UTC m=+1208.727519897" Oct 30 17:33:01.871071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:01.871040 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8848-bbb454"] Oct 30 17:33:01.875882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:01.875852 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8848-bbb454"] Oct 30 17:33:02.942352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:02.942317 4622 generic.go:356] "Generic (PLEG): container finished" podID="ccf363b6-8107-4ed0-87a9-09184173039e" containerID="2f2a573c650dc3af4805c90bf596ad6109402e56e2c8b75b491b9504840df73b" exitCode=137 Oct 30 17:33:02.948736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:02.948702 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8848-bbb454" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" containerName="test-curl-8848-bbb454" containerID="cri-o://fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05" gracePeriod=2 Oct 30 17:33:03.767393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:03.767367 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-b68665" Oct 30 17:33:03.924851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:03.924757 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\" (UniqueName: \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\") pod \"ccf363b6-8107-4ed0-87a9-09184173039e\" (UID: \"ccf363b6-8107-4ed0-87a9-09184173039e\") " Oct 30 17:33:03.929305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:03.929252 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq" pod "ccf363b6-8107-4ed0-87a9-09184173039e" (UID: "ccf363b6-8107-4ed0-87a9-09184173039e"). InnerVolumeSpecName "kube-api-access-h7flq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:03.972288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:03.971145 4622 scope.go:122] "RemoveContainer" containerID="2f2a573c650dc3af4805c90bf596ad6109402e56e2c8b75b491b9504840df73b" Oct 30 17:33:03.972288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:03.971338 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-b68665" Oct 30 17:33:04.001630 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:04.001581 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-b68665\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" pod="opendatahub/test-curl-8848-b68665" Oct 30 17:33:04.027237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:04.027200 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7flq\" (UniqueName: \"kubernetes.io/projected/ccf363b6-8107-4ed0-87a9-09184173039e-kube-api-access-h7flq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:04.029409 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:04.029381 4622 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccf363b6_8107_4ed0_87a9_09184173039e.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:33:04.375379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:04.375349 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf363b6-8107-4ed0-87a9-09184173039e" path="/var/lib/kubelet/pods/ccf363b6-8107-4ed0-87a9-09184173039e/volumes" Oct 30 17:33:05.164149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.164124 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:05.246909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.246816 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\" (UniqueName: \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\") pod \"ef624704-f043-4f4c-8ee1-d64e4f95551a\" (UID: \"ef624704-f043-4f4c-8ee1-d64e4f95551a\") " Oct 30 17:33:05.249349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.249315 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd" pod "ef624704-f043-4f4c-8ee1-d64e4f95551a" (UID: "ef624704-f043-4f4c-8ee1-d64e4f95551a"). InnerVolumeSpecName "kube-api-access-7lrpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:05.348178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.348146 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrpd\" (UniqueName: \"kubernetes.io/projected/ef624704-f043-4f4c-8ee1-d64e4f95551a-kube-api-access-7lrpd\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:05.987999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.987961 4622 generic.go:356] "Generic (PLEG): container finished" podID="ef624704-f043-4f4c-8ee1-d64e4f95551a" containerID="fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05" exitCode=137 Oct 30 17:33:05.988213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.988023 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:05.988213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.988066 4622 scope.go:122] "RemoveContainer" containerID="fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05" Oct 30 17:33:06.000083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:05.999945 4622 scope.go:122] "RemoveContainer" containerID="fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05" Oct 30 17:33:06.000309 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:06.000286 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05\": container with ID starting with fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05 not found: ID does not exist" containerID="fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05" Oct 30 17:33:06.000391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:06.000316 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05"} err="failed to get container status \"fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05\": rpc error: code = NotFound desc = could not find container \"fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05\": container with ID starting with fc1f78cd6952a2c754b7f3daaf6cde85119c4bdddb32ea81a537401f1d3eca05 not found: ID does not exist" Oct 30 17:33:06.009860 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:06.009828 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8848-bbb454\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" pod="opendatahub/test-curl-8848-bbb454" Oct 30 17:33:06.375241 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:06.375208 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" path="/var/lib/kubelet/pods/ef624704-f043-4f4c-8ee1-d64e4f95551a/volumes" Oct 30 17:33:07.091384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.091347 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4"] Oct 30 17:33:07.092018 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.091999 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" containerName="test-curl-8848-bbb454" Oct 30 17:33:07.092140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.092019 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" containerName="test-curl-8848-bbb454" Oct 30 17:33:07.092209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.092174 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef624704-f043-4f4c-8ee1-d64e4f95551a" containerName="test-curl-8848-bbb454" Oct 30 17:33:07.097078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.097057 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.114185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.114150 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4"] Oct 30 17:33:07.121434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.121413 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:33:07.121548 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.121447 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:33:07.173651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.173618 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.173823 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.173774 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktppv\" (UniqueName: \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.173823 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.173813 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.275145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.275110 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktppv\" (UniqueName: \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.275145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.275149 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.275369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.275329 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.275996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.275967 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.277643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.277616 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.298062 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.298036 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktppv\" (UniqueName: \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\") pod \"maas-api-revoke-sub-54a5cb01-4xms4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.411366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.411293 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:07.718622 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:07.718593 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155c9ab_b99f_4508_be9b_b9583af577f4.slice/crio-3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f WatchSource:0}: Error finding container 3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f: Status 404 returned error can't find the container with id 3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f Oct 30 17:33:07.719524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:07.719501 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4"] Oct 30 17:33:08.004546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.004447 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" event={"ID":"6155c9ab-b99f-4508-be9b-b9583af577f4","Type":"ContainerStarted","Data":"3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f"} Oct 30 17:33:08.701997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.701960 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:33:08.707125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.707100 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.731765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.731735 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:33:08.795069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795040 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795083 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgjj\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795272 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795333 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795375 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795398 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795515 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795556 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.795586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.795571 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.823921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.823850 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:33:08.840554 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.840524 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.857812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.857775 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:33:08.896577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896538 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.896762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896604 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.896762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896662 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.896762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896689 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.896762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896719 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.896762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896744 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896765 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896875 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896955 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896962 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.896985 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897011 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897037 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897067 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897008 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897205 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897254 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgjj\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897294 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897343 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897454 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkj97\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.897622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897531 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.898143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897676 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.898143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.897694 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.899126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.899110 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.899195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.899178 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.925569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.925536 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.925736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.925720 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgjj\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:08.998783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.998748 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.998914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.998794 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.998959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.998905 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999025 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.998988 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999034 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999062 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999086 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkj97\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999116 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999132 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999231 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999404 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999572 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999690 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:08.999961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:08.999941 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.001300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.001255 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.001399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.001342 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.012990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.012969 4622 generic.go:356] "Generic (PLEG): container finished" podID="6155c9ab-b99f-4508-be9b-b9583af577f4" containerID="dc700b3c1843ea0c32db66896fc305b99a5572cc14181303a71f4acd8db7b7de" exitCode=0 Oct 30 17:33:09.013086 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.013047 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" event={"ID":"6155c9ab-b99f-4508-be9b-b9583af577f4","Type":"ContainerDied","Data":"dc700b3c1843ea0c32db66896fc305b99a5572cc14181303a71f4acd8db7b7de"} Oct 30 17:33:09.022217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.022188 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.022420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.022405 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkj97\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.034754 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.034734 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:09.162516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.162475 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:09.369151 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:09.369122 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ca7223_c3d5_40bb_be56_b7b4858e90df.slice/crio-f2c4b935ff388568ed913daaf3a9fc3c9d177dd324a77fd612be781aef9f990d WatchSource:0}: Error finding container f2c4b935ff388568ed913daaf3a9fc3c9d177dd324a77fd612be781aef9f990d: Status 404 returned error can't find the container with id f2c4b935ff388568ed913daaf3a9fc3c9d177dd324a77fd612be781aef9f990d Oct 30 17:33:09.369655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.369633 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:33:09.372015 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.371984 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:09.372143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.372057 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:09.372143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.372110 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:09.473847 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:09.473817 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417f2c41_c04e_44fd_87c5_77fb7b1d6de7.slice/crio-609adbde90a5ef3c78ca779b0427ee5640ce3aab8c17199aa46f2f59d13c73d5 WatchSource:0}: Error finding container 609adbde90a5ef3c78ca779b0427ee5640ce3aab8c17199aa46f2f59d13c73d5: Status 404 returned error can't find the container with id 609adbde90a5ef3c78ca779b0427ee5640ce3aab8c17199aa46f2f59d13c73d5 Oct 30 17:33:09.474270 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.474241 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:33:09.476999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.476958 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:09.477106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.477046 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:09.477106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:09.477093 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:10.041390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.033017 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" event={"ID":"417f2c41-c04e-44fd-87c5-77fb7b1d6de7","Type":"ContainerStarted","Data":"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255"} Oct 30 17:33:10.041390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.033085 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" event={"ID":"417f2c41-c04e-44fd-87c5-77fb7b1d6de7","Type":"ContainerStarted","Data":"609adbde90a5ef3c78ca779b0427ee5640ce3aab8c17199aa46f2f59d13c73d5"} Oct 30 17:33:10.081349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.058029 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" event={"ID":"41ca7223-c3d5-40bb-be56-b7b4858e90df","Type":"ContainerStarted","Data":"f18e5526fb557f68dfef73f4a0dcb74df6e24991a975a320f1aec31f425c02a2"} Oct 30 17:33:10.081349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.058090 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" event={"ID":"41ca7223-c3d5-40bb-be56-b7b4858e90df","Type":"ContainerStarted","Data":"f2c4b935ff388568ed913daaf3a9fc3c9d177dd324a77fd612be781aef9f990d"} Oct 30 17:33:10.094391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.085011 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:10.125715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.125397 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:10.125715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.125452 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:10.178296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.170063 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:10.223382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.192632 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:10.223382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.192691 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:10.285213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.285136 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podStartSLOduration=2.2851112430000002 podStartE2EDuration="2.285111243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:10.092045683 +0000 UTC m=+1217.869517650" watchObservedRunningTime="2026-10-30 17:33:10.285111243 +0000 UTC m=+1218.062583199" Oct 30 17:33:10.301686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.299891 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podStartSLOduration=2.299869562 podStartE2EDuration="2.299869562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:10.145795313 +0000 UTC m=+1217.923267282" watchObservedRunningTime="2026-10-30 17:33:10.299869562 +0000 UTC m=+1218.077341518" Oct 30 17:33:10.643765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.643736 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:10.735002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.734962 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\") pod \"6155c9ab-b99f-4508-be9b-b9583af577f4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " Oct 30 17:33:10.735177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.735121 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\" (UniqueName: \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\") pod \"6155c9ab-b99f-4508-be9b-b9583af577f4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " Oct 30 17:33:10.735288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.735251 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\") pod \"6155c9ab-b99f-4508-be9b-b9583af577f4\" (UID: \"6155c9ab-b99f-4508-be9b-b9583af577f4\") " Oct 30 17:33:10.736029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.736000 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca" pod "6155c9ab-b99f-4508-be9b-b9583af577f4" (UID: "6155c9ab-b99f-4508-be9b-b9583af577f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:10.810310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.800038 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca" pod "6155c9ab-b99f-4508-be9b-b9583af577f4" (UID: "6155c9ab-b99f-4508-be9b-b9583af577f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:10.812467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.812430 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv" pod "6155c9ab-b99f-4508-be9b-b9583af577f4" (UID: "6155c9ab-b99f-4508-be9b-b9583af577f4"). InnerVolumeSpecName "kube-api-access-ktppv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:10.837080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.837054 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:10.837080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.837081 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6155c9ab-b99f-4508-be9b-b9583af577f4-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:10.837298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.837092 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktppv\" (UniqueName: \"kubernetes.io/projected/6155c9ab-b99f-4508-be9b-b9583af577f4-kube-api-access-ktppv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:10.908237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.902600 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:33:10.908237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.904535 4622 cpu_manager.go:414] "containerMap: removing container" podUID="6155c9ab-b99f-4508-be9b-b9583af577f4" containerName="revoke-keys" Oct 30 17:33:10.908237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.904561 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6155c9ab-b99f-4508-be9b-b9583af577f4" containerName="revoke-keys" Oct 30 17:33:10.908237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.904778 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="6155c9ab-b99f-4508-be9b-b9583af577f4" containerName="revoke-keys" Oct 30 17:33:10.909124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.909078 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.939587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.939552 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:33:10.940451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.940424 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5hp\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.948590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.948568 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.948795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.948779 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.961841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.961812 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-84af0117-gw-openshift-default-dockercfg-jb7zh\"" Oct 30 17:33:10.962249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.962225 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.962499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.962480 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.962718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.962703 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.962934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.962918 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.963090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.963075 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:10.963250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:10.963235 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.043970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.043932 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:11.065029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.064988 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:11.081975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.081929 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" Oct 30 17:33:11.085546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.082350 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4" event={"ID":"6155c9ab-b99f-4508-be9b-b9583af577f4","Type":"ContainerDied","Data":"3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f"} Oct 30 17:33:11.088705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.088671 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b34fda0554e758021c6738c8302cb00041a08a8aac4a92288b40ddeb136d99f" Oct 30 17:33:11.130994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.130955 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.145469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.145419 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.156284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.145753 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.156568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.156540 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.156772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.156750 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5hp\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.156912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.156896 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.157021 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.157005 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.157392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.157366 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.157557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.157541 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.172553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.158189 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.173089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.173062 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:11.173282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.173239 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:11.188286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.174403 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.189099 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.174591 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.217855 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.200543 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.217855 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.200794 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.222578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.222549 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.228482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.228254 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.234340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.229883 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5hp\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.285340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.284933 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\") pod \"e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.311541 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.311497 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:11.719050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.719019 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp"] Oct 30 17:33:11.831435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.831403 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:11.832826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.832776 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp"] Oct 30 17:33:11.898885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.873913 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:33:11.898885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.874599 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:33:11.925650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.925612 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:11.925831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.925660 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f699n\" (UniqueName: \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:11.925875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:11.925840 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.027499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.027407 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.027499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.027456 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f699n\" (UniqueName: \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.027872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.027850 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.028048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.028026 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.030659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.030633 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.037601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.036150 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:12.037601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.036191 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:12.040227 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:12.040192 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd932ab_8daa_4f48_8b79_8669dd665f3d.slice/crio-6c0b89c70b3223d63b693dbdddc2c9e63005f12a75fced3e46bb68da0b2fe081 WatchSource:0}: Error finding container 6c0b89c70b3223d63b693dbdddc2c9e63005f12a75fced3e46bb68da0b2fe081: Status 404 returned error can't find the container with id 6c0b89c70b3223d63b693dbdddc2c9e63005f12a75fced3e46bb68da0b2fe081 Oct 30 17:33:12.042307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.041169 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:33:12.067313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.067277 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f699n\" (UniqueName: \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\") pod \"maas-api-revoke-sub-d2966cb1-n4ppp\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.094815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.093904 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:12.094815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.094006 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:12.094815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.094051 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:12.180320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.171377 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:12.180320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.171437 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:12.210231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.210195 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:12.224525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.224489 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerStarted","Data":"6c0b89c70b3223d63b693dbdddc2c9e63005f12a75fced3e46bb68da0b2fe081"} Oct 30 17:33:12.722044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:12.718154 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp"] Oct 30 17:33:13.035610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.035573 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:13.035796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.035627 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:13.173286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.166135 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:13.173286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.166195 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:13.234628 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.234589 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" event={"ID":"aff15c34-5f2d-4ca6-8834-d7ac8327ab52","Type":"ContainerStarted","Data":"47effb947e4ec8e105a9747e8ae8f4d6b9cba4c03a8aaeb4322369f83ef392a5"} Oct 30 17:33:13.236563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.236542 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerStarted","Data":"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866"} Oct 30 17:33:13.287344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.286916 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podStartSLOduration=3.286895324 podStartE2EDuration="3.286895324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:13.284985068 +0000 UTC m=+1221.062457060" watchObservedRunningTime="2026-10-30 17:33:13.286895324 +0000 UTC m=+1221.064367281" Oct 30 17:33:13.317393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.317368 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:13.322592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:13.322568 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:14.035833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.035810 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:14.036033 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.036010 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:14.176715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.176681 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:14.177404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.177362 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:14.301285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.298443 4622 generic.go:356] "Generic (PLEG): container finished" podID="aff15c34-5f2d-4ca6-8834-d7ac8327ab52" containerID="156d9c918cb368882b4449c97052951c683ec99b738c2fe2f7e1298797677ed5" exitCode=0 Oct 30 17:33:14.301285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.300070 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" event={"ID":"aff15c34-5f2d-4ca6-8834-d7ac8327ab52","Type":"ContainerDied","Data":"156d9c918cb368882b4449c97052951c683ec99b738c2fe2f7e1298797677ed5"} Oct 30 17:33:14.301285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.300140 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:14.301285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:14.301152 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:33:15.035738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.035708 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:15.035900 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.035758 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:15.168291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.167573 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:15.168291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.167657 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:15.544285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.540243 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:15.605812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.605728 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\" (UniqueName: \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\") pod \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " Oct 30 17:33:15.606310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.606249 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\") pod \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " Oct 30 17:33:15.606508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.606489 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\") pod \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\" (UID: \"aff15c34-5f2d-4ca6-8834-d7ac8327ab52\") " Oct 30 17:33:15.607033 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.607006 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca" pod "aff15c34-5f2d-4ca6-8834-d7ac8327ab52" (UID: "aff15c34-5f2d-4ca6-8834-d7ac8327ab52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:15.607250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.607238 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:15.608296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.608271 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n" pod "aff15c34-5f2d-4ca6-8834-d7ac8327ab52" (UID: "aff15c34-5f2d-4ca6-8834-d7ac8327ab52"). InnerVolumeSpecName "kube-api-access-f699n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:15.608576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.608560 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca" pod "aff15c34-5f2d-4ca6-8834-d7ac8327ab52" (UID: "aff15c34-5f2d-4ca6-8834-d7ac8327ab52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:15.708845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.708806 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:15.708845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:15.708835 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f699n\" (UniqueName: \"kubernetes.io/projected/aff15c34-5f2d-4ca6-8834-d7ac8327ab52-kube-api-access-f699n\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:33:16.035244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.035212 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:33:16.035441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.035280 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 17:33:16.163688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.163655 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:16.175022 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.174982 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:16.384355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.370065 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" Oct 30 17:33:16.461334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.432256 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp" event={"ID":"aff15c34-5f2d-4ca6-8834-d7ac8327ab52","Type":"ContainerDied","Data":"47effb947e4ec8e105a9747e8ae8f4d6b9cba4c03a8aaeb4322369f83ef392a5"} Oct 30 17:33:16.468850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.468820 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47effb947e4ec8e105a9747e8ae8f4d6b9cba4c03a8aaeb4322369f83ef392a5" Oct 30 17:33:16.997797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.994136 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 17:33:16.997797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.995170 4622 cpu_manager.go:414] "containerMap: removing container" podUID="aff15c34-5f2d-4ca6-8834-d7ac8327ab52" containerName="revoke-keys" Oct 30 17:33:16.997797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.995193 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff15c34-5f2d-4ca6-8834-d7ac8327ab52" containerName="revoke-keys" Oct 30 17:33:16.997797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:16.995382 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff15c34-5f2d-4ca6-8834-d7ac8327ab52" containerName="revoke-keys" Oct 30 17:33:17.001603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.001578 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.017092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.016712 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 17:33:17.024087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.024060 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-506124\"/\"distinct-2-w2-506124-kserve-self-signed-certs\"" Oct 30 17:33:17.113282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.096031 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.113683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.113651 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.113994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.113972 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4cl\" (UniqueName: \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.114106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.114093 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.114255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.114242 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.121292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.114389 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.163507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.163479 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:33:17.163705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.163680 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 17:33:17.222644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.222602 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4cl\" (UniqueName: \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.223254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.223222 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.223566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.223546 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.223733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.223717 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.223916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.223894 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.224055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.224038 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.224560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.224509 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.224696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.224657 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.224942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.224923 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.235304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.227376 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.239964 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.239901 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.265332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.261026 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4cl\" (UniqueName: \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\") pod \"distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.342182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.336624 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:17.428594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.428561 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:17.430402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.430380 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:17.431854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.431836 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:33:17.741657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:17.741628 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 17:33:18.174772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.174730 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:18.175649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.175628 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:18.176230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.176200 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:33:18.409307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.409252 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" event={"ID":"5c06b9a2-2375-4173-afd0-7049113c75e5","Type":"ContainerStarted","Data":"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559"} Oct 30 17:33:18.409472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.409316 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" event={"ID":"5c06b9a2-2375-4173-afd0-7049113c75e5","Type":"ContainerStarted","Data":"90a2f91cc654be7ff5840d71141a8e1d04263e49796e0f40b5a41bdd1a2eab72"} Oct 30 17:33:18.450963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:18.450858 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podStartSLOduration=2.450844341 podStartE2EDuration="2.450844341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:18.449739825 +0000 UTC m=+1226.227211783" watchObservedRunningTime="2026-10-30 17:33:18.450844341 +0000 UTC m=+1226.228316296" Oct 30 17:33:19.337369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:19.337332 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:22.126287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.112349 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 17:33:22.141285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.127366 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:22.401637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.401535 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.005953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.005896 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.010759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.010720 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:33:23.209049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.887366 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.209049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.477046 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:33:23.209049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.093384 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 17:33:23.209049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.128125 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.209049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:22.468340 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:23.215524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.215473 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:33:23.441285 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:23.435178 4622 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.062s" Oct 30 17:33:23.441686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.441659 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:33:23.445285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.441843 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:33:23.445458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.445441 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:33:23.481191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.481120 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:33:23.867173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.867142 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:33:23.871268 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.871241 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.887567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.887532 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.887720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.887607 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.887897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.887875 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.887969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.887912 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.888042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.887993 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.888108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.888063 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.888153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.888119 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.888204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.888173 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smrj\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.888251 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.888217 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.893569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.893546 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:33:23.989094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989065 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989112 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989143 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989173 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989193 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989312 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989374 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989414 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.989492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.989475 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smrj\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.990075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.990045 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.990228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.990163 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.990228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.990167 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.990406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.990387 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.990578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.990551 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.991724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.991704 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:23.992167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:23.992139 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:24.012634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.012612 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smrj\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:24.012720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.012616 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:24.186872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.186792 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:24.517746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.517709 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:33:24.519960 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:24.519894 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb86078_c1be_4fd7_8dae_71231ed7b1ce.slice/crio-eea79513b8cb6f56a25cd6ce55065e6b2dd5c88f878dace89b5ebdb158de86e1 WatchSource:0}: Error finding container eea79513b8cb6f56a25cd6ce55065e6b2dd5c88f878dace89b5ebdb158de86e1: Status 404 returned error can't find the container with id eea79513b8cb6f56a25cd6ce55065e6b2dd5c88f878dace89b5ebdb158de86e1 Oct 30 17:33:24.525892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.523820 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:24.525892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.523907 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:24.525892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:24.523955 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:33:24.555642 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:24.552461 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:33:25.114811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.114768 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" event={"ID":"bcb86078-c1be-4fd7-8dae-71231ed7b1ce","Type":"ContainerStarted","Data":"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e"} Oct 30 17:33:25.115011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.114830 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" event={"ID":"bcb86078-c1be-4fd7-8dae-71231ed7b1ce","Type":"ContainerStarted","Data":"eea79513b8cb6f56a25cd6ce55065e6b2dd5c88f878dace89b5ebdb158de86e1"} Oct 30 17:33:25.178765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.178705 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podStartSLOduration=2.178689351 podStartE2EDuration="2.178689351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:25.176742735 +0000 UTC m=+1232.954214690" watchObservedRunningTime="2026-10-30 17:33:25.178689351 +0000 UTC m=+1232.956161307" Oct 30 17:33:25.197063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.195453 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:25.197276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.197131 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:25.197276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.197190 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:25.887714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.887682 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:33:25.892143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.892121 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:25.915519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.915487 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjzs\" (UniqueName: \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:25.915721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.915692 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:25.916479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.916447 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:25.916709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.916691 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:25.919434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.919414 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f8ac7b58-serving-cert\"" Oct 30 17:33:25.926402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:25.926378 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:33:26.017764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.017729 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.017948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.017786 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.017948 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:26.017884 4622 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-serving-cert: secret "maas-api-e2e-ait-f8ac7b58-serving-cert" not found Oct 30 17:33:26.017948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.017911 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.017948 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:26.017945 4622 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls podName:275a7fe4-f1b8-4347-8575-d4318e4c8a18 nodeName:}" failed. No retries permitted until 2026-10-30 17:33:26.517929523 +0000 UTC m=+1234.295401456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls") pod "maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" (UID: "275a7fe4-f1b8-4347-8575-d4318e4c8a18") : secret "maas-api-e2e-ait-f8ac7b58-serving-cert" not found Oct 30 17:33:26.018154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.017986 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjzs\" (UniqueName: \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.018682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.018655 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.020585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.020565 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.065933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.065902 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjzs\" (UniqueName: \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.187995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.187910 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:26.188175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.187980 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:26.526801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.526703 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.529350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.529317 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") pod \"maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:26.809276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:26.809226 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:27.180283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:27.178666 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:33:27.188905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:27.188737 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:27.188905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:27.188801 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:27.338058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:27.338024 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:27.397283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:27.396526 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:28.174716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.174678 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" event={"ID":"275a7fe4-f1b8-4347-8575-d4318e4c8a18","Type":"ContainerStarted","Data":"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a"} Oct 30 17:33:28.174892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.174877 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:28.174944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.174923 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" event={"ID":"275a7fe4-f1b8-4347-8575-d4318e4c8a18","Type":"ContainerStarted","Data":"aed9d0174856fd2832c213193ca6f926e51e76d58e3bd6a4232b27a2f4302225"} Oct 30 17:33:28.187147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.187120 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:28.187642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.187164 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:28.187642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.187193 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:28.218405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:28.218357 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" podStartSLOduration=2.812642521 podStartE2EDuration="3.218345024s" totalImagesPullingTime="405.702503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:28.215888882 +0000 UTC m=+1235.993360837" watchObservedRunningTime="2026-10-30 17:33:28.218345024 +0000 UTC m=+1235.995816978" Oct 30 17:33:29.187119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:29.187083 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:29.187400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:29.187135 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:30.187184 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:30.187145 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:30.187380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:30.187215 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:31.187760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:31.187728 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:33:31.188171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:31.187777 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 17:33:32.191495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:32.191405 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:32.191969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:32.191834 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:32.192494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:32.192473 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:33:34.188299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:34.188244 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:33:38.419157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.419127 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:33:38.423459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.423443 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.443955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.443926 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:33:38.444776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.444748 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-15a954ad-serving-cert\"" Oct 30 17:33:38.532451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.532410 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.532637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.532517 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.532637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.532583 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5d7\" (UniqueName: \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.532798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.532779 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.640649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.640607 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.643822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.643793 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.644507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.644486 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.645426 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.645345 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.645543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.645408 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5d7\" (UniqueName: \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.647577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.647551 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.649913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.649886 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.670513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.670441 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5d7\" (UniqueName: \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\") pod \"maas-api-e2e-derive-15a954ad-569bf88d79-nb668\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:38.739378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:38.739342 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:39.050626 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:33:39.050590 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660ddada_c42e_4698_82a4_3d24c066e561.slice/crio-53caaffc59d972b4b0cd46e730460a03cf2c10aa8350a841be88a9e5494a465d WatchSource:0}: Error finding container 53caaffc59d972b4b0cd46e730460a03cf2c10aa8350a841be88a9e5494a465d: Status 404 returned error can't find the container with id 53caaffc59d972b4b0cd46e730460a03cf2c10aa8350a841be88a9e5494a465d Oct 30 17:33:39.051442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:39.051412 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:33:39.052513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:39.052497 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:33:39.281458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:39.281423 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" event={"ID":"660ddada-c42e-4698-82a4-3d24c066e561","Type":"ContainerStarted","Data":"53caaffc59d972b4b0cd46e730460a03cf2c10aa8350a841be88a9e5494a465d"} Oct 30 17:33:40.029949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.029918 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g"] Oct 30 17:33:40.043075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.043045 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef8426c-2nm7g"] Oct 30 17:33:40.292518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.292465 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" event={"ID":"660ddada-c42e-4698-82a4-3d24c066e561","Type":"ContainerStarted","Data":"9c71be9b1dea9bc966e5d5e0916695d1167d4913c7415d4125e6227af4320b50"} Oct 30 17:33:40.292811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.292543 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:40.340701 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.340642 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" podStartSLOduration=1.987469661 podStartE2EDuration="2.340619625s" totalImagesPullingTime="353.149964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:40.335546095 +0000 UTC m=+1248.113018052" watchObservedRunningTime="2026-10-30 17:33:40.340619625 +0000 UTC m=+1248.118091584" Oct 30 17:33:40.378495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:40.378464 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc87ed61-3e8b-4083-adfe-d30f38e1bbf2" path="/var/lib/kubelet/pods/dc87ed61-3e8b-4083-adfe-d30f38e1bbf2/volumes" Oct 30 17:33:46.319955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:46.319923 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:33:49.118964 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.118914 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:33:49.121173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.119685 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 17:33:49.201490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.199790 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.403106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.402356 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.512227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.512176 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.736343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.732686 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:49.736343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.733960 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:33:49.882192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:49.863524 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:33:50.134287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.131366 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.134287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.131738 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.134287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.133110 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:33:50.174638 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.174595 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.176059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.176033 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:33:50.239341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.200520 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.303307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.241154 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:50.757292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.745563 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:50.878285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:50.877959 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.040347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.023768 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:33:51.040347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.023863 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:33:51.138844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.138788 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:33:51.187134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.187095 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 17:33:51.216489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.216444 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 17:33:51.424315 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.424188 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.479876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.452015 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:33:51.794345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.742556 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 17:33:51.798284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.794593 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:33:51.878378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.878343 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:33:51.878658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.878629 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:51.878761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:51.878391 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.197223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.197124 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.198460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.198431 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:33:52.218364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.218327 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.466965 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.466871 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded" Oct 30 17:33:52.503289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.501803 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.503902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.503870 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:33:52.535256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.535225 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:33:52.541283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.535558 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:33:52.797294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.797247 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 17:33:52.811285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.799454 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 17:33:52.976927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.865034 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:52.977972 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.977944 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:33:52.987332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:52.947309 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:33:53.097425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.097332 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:53.317595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.216851 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:33:53.318961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.217941 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 17:33:53.319444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.220477 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:33:53.319702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.319685 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:33:53.320273 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.320237 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:33:53.411353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.408667 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:33:53.412127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.412106 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:33:53.487283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.465812 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 17:33:53.487283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:53.479028 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:33:54.535145 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:33:54.535109 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:33:59.888185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:59.885836 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:32790->10.134.0.90:15021: read: connection reset by peer" start-of-body= Oct 30 17:33:59.888185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:33:59.885905 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:32790->10.134.0.90:15021: read: connection reset by peer" Oct 30 17:34:00.323114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.323074 4622 generic.go:356] "Generic (PLEG): container finished" podID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerID="4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866" exitCode=0 Oct 30 17:34:00.323114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.323118 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerDied","Data":"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866"} Oct 30 17:34:00.323528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.323508 4622 scope.go:122] "RemoveContainer" containerID="4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866" Oct 30 17:34:00.324936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.324908 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:00.324994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.324978 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:00.325028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:00.325012 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:01.333796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:01.333757 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerStarted","Data":"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82"} Oct 30 17:34:02.312623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:02.312586 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:02.312813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:02.312789 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:34:02.312859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:02.312840 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:34:03.312225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:03.312194 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:34:03.312652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:03.312248 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:34:04.035949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:04.035909 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh"] Oct 30 17:34:04.069863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:04.069827 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b30573-6dlrh"] Oct 30 17:34:04.312641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:04.312550 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:34:04.312641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:04.312611 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:34:04.387293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:04.387222 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e36521d-025c-4964-9f41-d0b399495eb5" path="/var/lib/kubelet/pods/3e36521d-025c-4964-9f41-d0b399495eb5/volumes" Oct 30 17:34:05.098285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:05.092135 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk"] Oct 30 17:34:05.104293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:05.103363 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95c0bb8a-sfjbk"] Oct 30 17:34:05.312949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:05.312914 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:34:05.313365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:05.312962 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:34:06.312077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:06.312041 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:34:06.312275 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:06.312112 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 17:34:06.377733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:06.377699 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82" path="/var/lib/kubelet/pods/f58dbbc5-6bfe-4d2a-a9dc-b37d61ed4e82/volumes" Oct 30 17:34:07.316465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:07.316435 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:07.316938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:07.316922 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:07.317570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:07.317549 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:09.005856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.005822 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:34:09.009448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.009430 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.034939 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.034909 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:34:09.111077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111044 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111197 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111247 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111393 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111428 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111532 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111559 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111579 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.111728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.111716 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9kv\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213446 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9kv\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213534 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213631 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213665 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213748 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213865 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.213954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213937 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214013 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213965 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214013 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.213985 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.214123 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.214142 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.214419 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.214484 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.214668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.214636 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.216042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.216019 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.216299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.216279 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.236540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.236514 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.236706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.236690 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9kv\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\") pod \"e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.322535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.322502 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:09.643990 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:34:09.643962 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a23bd3_25d9_43ad_8cc8_b0e3c6d7b3d5.slice/crio-33c6f4ace35af8df37f1e56f6f29ac3a5d7645fc6784b4bafe1a20fa2866a5fb WatchSource:0}: Error finding container 33c6f4ace35af8df37f1e56f6f29ac3a5d7645fc6784b4bafe1a20fa2866a5fb: Status 404 returned error can't find the container with id 33c6f4ace35af8df37f1e56f6f29ac3a5d7645fc6784b4bafe1a20fa2866a5fb Oct 30 17:34:09.645072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.645044 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:34:09.646650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.646613 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:09.646727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.646697 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:09.646781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:09.646742 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:10.445404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:10.445367 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" event={"ID":"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5","Type":"ContainerStarted","Data":"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d"} Oct 30 17:34:10.445796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:10.445419 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" event={"ID":"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5","Type":"ContainerStarted","Data":"33c6f4ace35af8df37f1e56f6f29ac3a5d7645fc6784b4bafe1a20fa2866a5fb"} Oct 30 17:34:10.521608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:10.521544 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podStartSLOduration=2.521526211 podStartE2EDuration="2.521526211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:10.502955199 +0000 UTC m=+1278.280427154" watchObservedRunningTime="2026-10-30 17:34:10.521526211 +0000 UTC m=+1278.298998168" Oct 30 17:34:11.323170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:11.323136 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:11.324713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:11.324688 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:11.324842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:11.324733 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:11.881490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:11.881442 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:34:11.882311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:11.882277 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" podUID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" containerName="maas-api" containerID="cri-o://a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a" gracePeriod=30 Oct 30 17:34:12.335067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.326216 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:12.335729 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.335694 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:12.363523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.363490 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:34:12.406327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.399735 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\" (UniqueName: \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\") pod \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " Oct 30 17:34:12.406327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.399885 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") pod \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " Oct 30 17:34:12.406327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.399942 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\") pod \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " Oct 30 17:34:12.406327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.400187 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\") pod \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\" (UID: \"275a7fe4-f1b8-4347-8575-d4318e4c8a18\") " Oct 30 17:34:12.407031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.406925 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca" pod "275a7fe4-f1b8-4347-8575-d4318e4c8a18" (UID: "275a7fe4-f1b8-4347-8575-d4318e4c8a18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:12.422666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.422590 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs" pod "275a7fe4-f1b8-4347-8575-d4318e4c8a18" (UID: "275a7fe4-f1b8-4347-8575-d4318e4c8a18"). InnerVolumeSpecName "kube-api-access-fbjzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:12.459440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.459375 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs" pod "275a7fe4-f1b8-4347-8575-d4318e4c8a18" (UID: "275a7fe4-f1b8-4347-8575-d4318e4c8a18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:12.467305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.464416 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls" pod "275a7fe4-f1b8-4347-8575-d4318e4c8a18" (UID: "275a7fe4-f1b8-4347-8575-d4318e4c8a18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:12.520778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.520750 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.521183 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.521166 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275a7fe4-f1b8-4347-8575-d4318e4c8a18-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.521431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.521415 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjzs\" (UniqueName: \"kubernetes.io/projected/275a7fe4-f1b8-4347-8575-d4318e4c8a18-kube-api-access-fbjzs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.521667 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.521654 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275a7fe4-f1b8-4347-8575-d4318e4c8a18-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.554353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.553634 4622 generic.go:356] "Generic (PLEG): container finished" podID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" containerID="a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a" exitCode=0 Oct 30 17:34:12.588688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.554771 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" Oct 30 17:34:12.588688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.555370 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" event={"ID":"275a7fe4-f1b8-4347-8575-d4318e4c8a18","Type":"ContainerDied","Data":"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a"} Oct 30 17:34:12.588688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.555422 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng" event={"ID":"275a7fe4-f1b8-4347-8575-d4318e4c8a18","Type":"ContainerDied","Data":"aed9d0174856fd2832c213193ca6f926e51e76d58e3bd6a4232b27a2f4302225"} Oct 30 17:34:12.588688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.555454 4622 scope.go:122] "RemoveContainer" containerID="a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a" Oct 30 17:34:12.778287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.767584 4622 scope.go:122] "RemoveContainer" containerID="a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a" Oct 30 17:34:12.801851 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:34:12.801813 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a\": container with ID starting with a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a not found: ID does not exist" containerID="a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a" Oct 30 17:34:12.802094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.802069 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a"} err="failed to get container status \"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a\": rpc error: code = NotFound desc = could not find container \"a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a\": container with ID starting with a20e5694aa3b4aae72966a61fc201be801a49d028fa4aa213aaa7cdc6502398a not found: ID does not exist" Oct 30 17:34:12.814431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.814339 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:34:12.820750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:12.820725 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f8ac7b58-77fff6fdd6-ndnng"] Oct 30 17:34:13.328282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:13.327832 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:13.328282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:13.327916 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:14.323030 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:14.322987 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:14.323233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:14.323048 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:14.506970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:14.506933 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" path="/var/lib/kubelet/pods/275a7fe4-f1b8-4347-8575-d4318e4c8a18/volumes" Oct 30 17:34:15.323700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:15.323658 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:15.323917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:15.323758 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:16.323161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:16.323122 4622 patch_prober.go:29] interesting pod/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:34:16.323721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:16.323185 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 17:34:17.831104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:17.831072 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:17.831594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:17.831421 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:17.832414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:17.832391 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:34:19.729642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.729607 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 17:34:19.730712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.730687 4622 cpu_manager.go:414] "containerMap: removing container" podUID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" containerName="maas-api" Oct 30 17:34:19.730712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.730713 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" containerName="maas-api" Oct 30 17:34:19.730895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.730881 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="275a7fe4-f1b8-4347-8575-d4318e4c8a18" containerName="maas-api" Oct 30 17:34:19.734810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.734785 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.752978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.752947 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 17:34:19.759510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.759476 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-306aed\"/\"distinct-2-w3-306aed-kserve-self-signed-certs\"" Oct 30 17:34:19.771654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.771627 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.771791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.771775 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.771871 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.771853 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz862\" (UniqueName: \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.771931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.771903 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.772010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.771997 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.772067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.772037 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.873872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.873841 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.873997 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz862\" (UniqueName: \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874085 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874207 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874167 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874204 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874222 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874415 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874514 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874488 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.874625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.874602 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.877070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.877041 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.877188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.877145 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.905940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.905904 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz862\" (UniqueName: \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\") pod \"distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:19.945065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.945026 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:34:19.945591 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:19.945537 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" containerID="cri-o://6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82" gracePeriod=30 Oct 30 17:34:20.052617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:20.052582 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:20.571694 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:34:20.571639 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c442a99_d65a_49df_a1cf_79938df4f4ac.slice/crio-20bf41e1ad935bf53013e744c497b100cbff0de32827c4dc84f376c352e2ece8 WatchSource:0}: Error finding container 20bf41e1ad935bf53013e744c497b100cbff0de32827c4dc84f376c352e2ece8: Status 404 returned error can't find the container with id 20bf41e1ad935bf53013e744c497b100cbff0de32827c4dc84f376c352e2ece8 Oct 30 17:34:20.572308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:20.572240 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 17:34:20.658359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:20.658316 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" event={"ID":"5c442a99-d65a-49df-a1cf-79938df4f4ac","Type":"ContainerStarted","Data":"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de"} Oct 30 17:34:20.658487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:20.658373 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" event={"ID":"5c442a99-d65a-49df-a1cf-79938df4f4ac","Type":"ContainerStarted","Data":"20bf41e1ad935bf53013e744c497b100cbff0de32827c4dc84f376c352e2ece8"} Oct 30 17:34:20.704176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:20.704122 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podStartSLOduration=1.704107959 podStartE2EDuration="1.704107959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:20.700388572 +0000 UTC m=+1288.477860527" watchObservedRunningTime="2026-10-30 17:34:20.704107959 +0000 UTC m=+1288.481579910" Oct 30 17:34:21.058296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:21.055455 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:22.339756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:22.339712 4622 patch_prober.go:29] interesting pod/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:34:22.340250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:22.339777 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:34:23.187456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.187384 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:34:23.187652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.187491 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:23.541270 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.541216 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:34:23.542586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.542135 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" podUID="660ddada-c42e-4698-82a4-3d24c066e561" containerName="maas-api" containerID="cri-o://9c71be9b1dea9bc966e5d5e0916695d1167d4913c7415d4125e6227af4320b50" gracePeriod=30 Oct 30 17:34:23.833372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.833233 4622 generic.go:356] "Generic (PLEG): container finished" podID="660ddada-c42e-4698-82a4-3d24c066e561" containerID="9c71be9b1dea9bc966e5d5e0916695d1167d4913c7415d4125e6227af4320b50" exitCode=0 Oct 30 17:34:23.833531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.833372 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" event={"ID":"660ddada-c42e-4698-82a4-3d24c066e561","Type":"ContainerDied","Data":"9c71be9b1dea9bc966e5d5e0916695d1167d4913c7415d4125e6227af4320b50"} Oct 30 17:34:23.943433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:23.943409 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:34:24.088973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.088891 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\") pod \"660ddada-c42e-4698-82a4-3d24c066e561\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " Oct 30 17:34:24.088973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.088960 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\") pod \"660ddada-c42e-4698-82a4-3d24c066e561\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " Oct 30 17:34:24.089182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.089082 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\") pod \"660ddada-c42e-4698-82a4-3d24c066e561\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " Oct 30 17:34:24.089182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.089157 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\" (UniqueName: \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\") pod \"660ddada-c42e-4698-82a4-3d24c066e561\" (UID: \"660ddada-c42e-4698-82a4-3d24c066e561\") " Oct 30 17:34:24.089321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.089300 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca" pod "660ddada-c42e-4698-82a4-3d24c066e561" (UID: "660ddada-c42e-4698-82a4-3d24c066e561"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:24.089746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.089725 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/660ddada-c42e-4698-82a4-3d24c066e561-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:24.091127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.091096 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls" pod "660ddada-c42e-4698-82a4-3d24c066e561" (UID: "660ddada-c42e-4698-82a4-3d24c066e561"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:24.091242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.091131 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs" pod "660ddada-c42e-4698-82a4-3d24c066e561" (UID: "660ddada-c42e-4698-82a4-3d24c066e561"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:24.091242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.091161 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7" pod "660ddada-c42e-4698-82a4-3d24c066e561" (UID: "660ddada-c42e-4698-82a4-3d24c066e561"). InnerVolumeSpecName "kube-api-access-7g5d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:24.191413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.191376 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:24.191413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.191404 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g5d7\" (UniqueName: \"kubernetes.io/projected/660ddada-c42e-4698-82a4-3d24c066e561-kube-api-access-7g5d7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:24.191413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.191415 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/660ddada-c42e-4698-82a4-3d24c066e561-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:24.536071 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:34:24.536037 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:34:24.842679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.842647 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" event={"ID":"660ddada-c42e-4698-82a4-3d24c066e561","Type":"ContainerDied","Data":"53caaffc59d972b4b0cd46e730460a03cf2c10aa8350a841be88a9e5494a465d"} Oct 30 17:34:24.842679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.842684 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668" Oct 30 17:34:24.843190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.842715 4622 scope.go:122] "RemoveContainer" containerID="9c71be9b1dea9bc966e5d5e0916695d1167d4913c7415d4125e6227af4320b50" Oct 30 17:34:24.903397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.903363 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:34:24.916863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:24.916823 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15a954ad-569bf88d79-nb668"] Oct 30 17:34:25.283626 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.283600 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:25.417422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417329 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.417422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417397 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.417652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417464 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.417652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417516 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.417781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417715 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:25.417781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417762 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.417948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417809 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.418005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.417960 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.418061 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418002 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.418116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418093 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\") pod \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\" (UID: \"9fd932ab-8daa-4f48-8b79-8669dd665f3d\") " Oct 30 17:34:25.418434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418414 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:25.418521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418433 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:25.418642 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418620 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:25.418956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418933 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.419075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418957 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.419075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418969 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.419075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.418979 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.419252 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.419132 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:25.420649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.420622 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:25.420724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.420707 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:34:25.420985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.420932 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "kube-api-access-xc5hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:25.421084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.421041 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token" pod "9fd932ab-8daa-4f48-8b79-8669dd665f3d" (UID: "9fd932ab-8daa-4f48-8b79-8669dd665f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:25.520357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.520213 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.520357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.520246 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.520357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.520277 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.520357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.520291 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fd932ab-8daa-4f48-8b79-8669dd665f3d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.520357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.520304 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5hp\" (UniqueName: \"kubernetes.io/projected/9fd932ab-8daa-4f48-8b79-8669dd665f3d-kube-api-access-xc5hp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:34:25.858021 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.857987 4622 generic.go:356] "Generic (PLEG): container finished" podID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerID="6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82" exitCode=0 Oct 30 17:34:25.858505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.858037 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" Oct 30 17:34:25.858505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.858069 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerDied","Data":"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82"} Oct 30 17:34:25.858505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.858123 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8" event={"ID":"9fd932ab-8daa-4f48-8b79-8669dd665f3d","Type":"ContainerDied","Data":"6c0b89c70b3223d63b693dbdddc2c9e63005f12a75fced3e46bb68da0b2fe081"} Oct 30 17:34:25.858505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.858156 4622 scope.go:122] "RemoveContainer" containerID="6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82" Oct 30 17:34:25.870568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.870552 4622 scope.go:122] "RemoveContainer" containerID="4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866" Oct 30 17:34:25.882166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.882135 4622 scope.go:122] "RemoveContainer" containerID="6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82" Oct 30 17:34:25.882497 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:34:25.882475 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82\": container with ID starting with 6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82 not found: ID does not exist" containerID="6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82" Oct 30 17:34:25.882574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.882511 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82"} err="failed to get container status \"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82\": rpc error: code = NotFound desc = could not find container \"6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82\": container with ID starting with 6c5e1850e7fd779838fec7bb8b3352c1ba199a124ff3407667694cb6fd5d3e82 not found: ID does not exist" Oct 30 17:34:25.882574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.882531 4622 scope.go:122] "RemoveContainer" containerID="4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866" Oct 30 17:34:25.882864 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:34:25.882845 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866\": container with ID starting with 4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866 not found: ID does not exist" containerID="4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866" Oct 30 17:34:25.882924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.882869 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866"} err="failed to get container status \"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866\": rpc error: code = NotFound desc = could not find container \"4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866\": container with ID starting with 4488f6cfa4f53118d66744c85f8ef6053dd61846f0aa848a1013a7a681723866 not found: ID does not exist" Oct 30 17:34:25.915635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.915607 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:34:25.928667 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:25.928637 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-84af0117-gw-openshift-default-587d886b9-59lm8"] Oct 30 17:34:26.374567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:26.374533 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660ddada-c42e-4698-82a4-3d24c066e561" path="/var/lib/kubelet/pods/660ddada-c42e-4698-82a4-3d24c066e561/volumes" Oct 30 17:34:26.375019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:26.375005 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" path="/var/lib/kubelet/pods/9fd932ab-8daa-4f48-8b79-8669dd665f3d/volumes" Oct 30 17:34:30.053965 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:30.053924 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:30.066854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:30.066830 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:30.912383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:30.912352 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:36.621154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621115 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 17:34:36.621762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621746 4622 cpu_manager.go:414] "containerMap: removing container" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:36.621806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621763 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:36.621806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621776 4622 cpu_manager.go:414] "containerMap: removing container" podUID="660ddada-c42e-4698-82a4-3d24c066e561" containerName="maas-api" Oct 30 17:34:36.621806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621781 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660ddada-c42e-4698-82a4-3d24c066e561" containerName="maas-api" Oct 30 17:34:36.621918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621909 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:36.621953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621925 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:36.621953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.621933 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="660ddada-c42e-4698-82a4-3d24c066e561" containerName="maas-api" Oct 30 17:34:36.624487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.624463 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.642588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.642563 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 17:34:36.642702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.642675 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 17:34:36.642799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.642788 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 17:34:36.642799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.642793 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m72mh\"" Oct 30 17:34:36.661094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.661069 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 17:34:36.686292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686251 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxtkt\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686405 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686433 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686470 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686647 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686684 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.686849 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.686821 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.787966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.787923 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxtkt\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788036 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788061 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788100 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788242 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788292 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788368 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.788976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.788944 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.790649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.790621 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.790759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.790670 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.790854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.790833 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.790906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.790848 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.811561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.811536 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxtkt\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.813044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.813025 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kz8h\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:36.936312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:36.936218 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:37.252057 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:34:37.252018 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cf8876_b173_4519_8887_e9acf5c4005b.slice/crio-104c99cce5f8d01a61e8a63ce41ac2bfbe50df73dca71657cc9016cedcd390d2 WatchSource:0}: Error finding container 104c99cce5f8d01a61e8a63ce41ac2bfbe50df73dca71657cc9016cedcd390d2: Status 404 returned error can't find the container with id 104c99cce5f8d01a61e8a63ce41ac2bfbe50df73dca71657cc9016cedcd390d2 Oct 30 17:34:37.253293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:37.253242 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 17:34:37.964778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:37.964747 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" event={"ID":"b4cf8876-b173-4519-8887-e9acf5c4005b","Type":"ContainerStarted","Data":"104c99cce5f8d01a61e8a63ce41ac2bfbe50df73dca71657cc9016cedcd390d2"} Oct 30 17:34:38.978050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:38.978012 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:34:38.982099 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:38.982073 4622 cpu_manager.go:414] "containerMap: removing container" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:38.982294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:38.982275 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd932ab-8daa-4f48-8b79-8669dd665f3d" containerName="istio-proxy" Oct 30 17:34:38.988731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:38.988709 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:38.998930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:38.998904 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:34:39.126924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.126886 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.126935 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127025 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127091 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127175 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127227 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127257 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127359 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.127454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.127407 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228410 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228494 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228568 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228591 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228622 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228696 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228811 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228867 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.228912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.229115 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228945 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.229115 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228979 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.229115 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.228975 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.229232 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.229163 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.229674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.229648 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.237318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.231438 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.237318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.234749 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.252374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.252339 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.252374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.252365 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\") pod \"e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.317297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.317223 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:39.645325 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:34:39.645291 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fe9f14_a4be_46be_a29e_4b515fc17aa7.slice/crio-d78c79060ec6f0dbb52597cdeb9d1cf8c6ea5dec32ed5148dc842070244b1c76 WatchSource:0}: Error finding container d78c79060ec6f0dbb52597cdeb9d1cf8c6ea5dec32ed5148dc842070244b1c76: Status 404 returned error can't find the container with id d78c79060ec6f0dbb52597cdeb9d1cf8c6ea5dec32ed5148dc842070244b1c76 Oct 30 17:34:39.647018 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.646990 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:34:39.648199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.648167 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:39.648336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.648239 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:39.648336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.648294 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:39.986432 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.986394 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:39.986746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.986480 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:34:39.994895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.994868 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" event={"ID":"50fe9f14-a4be-46be-a29e-4b515fc17aa7","Type":"ContainerStarted","Data":"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3"} Oct 30 17:34:39.995015 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:39.994924 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" event={"ID":"50fe9f14-a4be-46be-a29e-4b515fc17aa7","Type":"ContainerStarted","Data":"d78c79060ec6f0dbb52597cdeb9d1cf8c6ea5dec32ed5148dc842070244b1c76"} Oct 30 17:34:40.318366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:40.318325 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:40.320274 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:40.320233 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:40.320446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:40.320329 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:41.020426 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.020393 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" event={"ID":"b4cf8876-b173-4519-8887-e9acf5c4005b","Type":"ContainerStarted","Data":"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc"} Oct 30 17:34:41.021418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.021391 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:41.023085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.023056 4622 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5kz8h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:34:41.023202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.023101 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:34:41.110477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.110416 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" podStartSLOduration=2.37993353 podStartE2EDuration="5.110393933s" totalImagesPullingTime="2.730460403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:41.095593044 +0000 UTC m=+1308.873064993" watchObservedRunningTime="2026-10-30 17:34:41.110393933 +0000 UTC m=+1308.887865889" Oct 30 17:34:41.110764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.110736 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podStartSLOduration=3.110731063 podStartE2EDuration="3.110731063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:40.050795822 +0000 UTC m=+1307.828267777" watchObservedRunningTime="2026-10-30 17:34:41.110731063 +0000 UTC m=+1308.888203019" Oct 30 17:34:41.318247 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.318208 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:41.318452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:41.318275 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:42.026796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:42.026766 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 17:34:42.318332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:42.318257 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:42.318510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:42.318360 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:43.317517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:43.317484 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:43.318001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:43.317547 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:44.318004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:44.317966 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:44.318417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:44.318037 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:45.318486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:45.318444 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:45.318873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:45.318504 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:46.317731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:46.317690 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:34:46.317905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:46.317761 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 17:34:47.321714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:47.321686 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:47.322170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:47.322150 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:47.322797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:47.322778 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:34:52.843679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:52.843637 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:52.948469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:52.948420 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:52.994441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:52.994382 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.267467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.267341 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 17:34:53.527352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.526971 4622 scope.go:122] "RemoveContainer" containerID="6185f08d706cf8cb888cd7376667c2c552bc4fa934ebd6d384d45263d0dea1b3" Oct 30 17:34:53.550779 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.540205 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:34:53.853852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.853385 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 17:34:53.887938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.887880 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:34:53.914386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.914339 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.949429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.949366 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:53.950115 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:53.950090 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:54.176598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:54.176514 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:34:54.275304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:54.273344 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:34:54.275304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:54.274200 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:34:54.304819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:34:54.304790 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:34:54.616315 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:34:54.616245 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:35:24.537137 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:35:24.537103 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:35:53.734189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:35:53.734117 4622 scope.go:122] "RemoveContainer" containerID="fb902a09274d2386a9a1e9ffaa68755accf70d11afd29586eea8154040b753fb" Oct 30 17:35:53.744187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:35:53.744164 4622 scope.go:122] "RemoveContainer" containerID="379bd115ae8ca25e1333c5cb7495018a6bd1fac427d3ee645bed95bd2e1cec0a" Oct 30 17:35:54.534635 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:35:54.534600 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:36:01.783799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.783769 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 17:36:01.788201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.788184 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.821335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.821297 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-c31c73-serving-cert\"" Oct 30 17:36:01.832481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.832454 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 17:36:01.879625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.879594 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.879790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.879731 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvm8k\" (UniqueName: \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.879847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.879819 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.880135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.880119 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.981373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.981337 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.981602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.981583 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.981672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.981657 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.981746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.981729 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvm8k\" (UniqueName: \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.982250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.982227 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.983793 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.983772 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:01.983929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:01.983912 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:02.005876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:02.005852 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvm8k\" (UniqueName: \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\") pod \"maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:02.102333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:02.102225 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:02.431642 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:36:02.431612 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eca3c38_3a21_4781_bbb2_03e8a27c9e88.slice/crio-bb2f6fc8c038dbe274bc3b4e0b898b728645e11a749f16a206df8c6df6d62082 WatchSource:0}: Error finding container bb2f6fc8c038dbe274bc3b4e0b898b728645e11a749f16a206df8c6df6d62082: Status 404 returned error can't find the container with id bb2f6fc8c038dbe274bc3b4e0b898b728645e11a749f16a206df8c6df6d62082 Oct 30 17:36:02.433250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:02.433225 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 17:36:02.896091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:02.896048 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" event={"ID":"6eca3c38-3a21-4781-bbb2-03e8a27c9e88","Type":"ContainerStarted","Data":"bb2f6fc8c038dbe274bc3b4e0b898b728645e11a749f16a206df8c6df6d62082"} Oct 30 17:36:03.906710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:03.906664 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" event={"ID":"6eca3c38-3a21-4781-bbb2-03e8a27c9e88","Type":"ContainerStarted","Data":"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b"} Oct 30 17:36:03.907322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:03.906846 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:09.919904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:09.919877 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 17:36:09.981440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:09.981380 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" podStartSLOduration=8.591127437 podStartE2EDuration="8.981364677s" totalImagesPullingTime="390.23724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:03.963580296 +0000 UTC m=+1391.741052253" watchObservedRunningTime="2026-10-30 17:36:09.981364677 +0000 UTC m=+1397.758836641" Oct 30 17:36:17.811694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.811662 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf"] Oct 30 17:36:17.816105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.816075 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:17.836209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.836178 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf"] Oct 30 17:36:17.842456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.842429 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:36:17.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.842518 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:36:17.967975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.967938 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:17.968149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.968117 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94tj\" (UniqueName: \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:17.968324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:17.968309 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.070109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.070027 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.070109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.070101 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.070394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.070201 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94tj\" (UniqueName: \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.070825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.070800 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.072671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.072647 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.095996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.095973 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94tj\" (UniqueName: \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\") pod \"maas-api-revoke-sub-3277aa70-nfzbf\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.136027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.136001 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:18.490892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:18.479941 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf"] Oct 30 17:36:19.028412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.028331 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" event={"ID":"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db","Type":"ContainerStarted","Data":"78295f22f3075ce7a3fcc54e15e7fb1d697bb61302a8d27bad7c84509acf8eb8"} Oct 30 17:36:19.028412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.028404 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" event={"ID":"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db","Type":"ContainerStarted","Data":"943a7ba83048a7af9026e8d2200339b9c4fae3acb6821ec4436ceda902e4c742"} Oct 30 17:36:19.088286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.088223 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" podStartSLOduration=2.088208848 podStartE2EDuration="2.088208848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:19.083081554 +0000 UTC m=+1406.860553510" watchObservedRunningTime="2026-10-30 17:36:19.088208848 +0000 UTC m=+1406.865680802" Oct 30 17:36:19.515852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.515820 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:36:19.521136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.521111 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.572868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.570577 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-df8b20-openshift-default-dockercfg-mtnwq\"" Oct 30 17:36:19.584325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.577535 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:36:19.616246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.616208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.616539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.616516 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.616709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.616690 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.616947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.616928 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.617155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.617138 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmg6l\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.617334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.617307 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.617429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.617389 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.617497 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.617472 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.617556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.617518 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.720252 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.719516 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724373 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmg6l\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724417 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724476 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724538 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724567 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724674 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724734 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.724812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.724800 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.725155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.725130 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.725257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.720097 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.725407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.725389 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.726973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.726047 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.729447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.728590 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.741281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.737257 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.741281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.737600 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.765211 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.765116 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmg6l\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.792885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.792850 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\") pod \"e2e-api-a-df8b20-openshift-default-6598475757-9xfbh\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:19.883547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:19.882923 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:20.077397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.077363 4622 generic.go:356] "Generic (PLEG): container finished" podID="4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" containerID="78295f22f3075ce7a3fcc54e15e7fb1d697bb61302a8d27bad7c84509acf8eb8" exitCode=0 Oct 30 17:36:20.077952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.077913 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" event={"ID":"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db","Type":"ContainerDied","Data":"78295f22f3075ce7a3fcc54e15e7fb1d697bb61302a8d27bad7c84509acf8eb8"} Oct 30 17:36:20.507041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.506999 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:36:20.507596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.507571 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:36:20.507784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.507764 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:36:20.523293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:20.521650 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:36:21.153354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.152253 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerStarted","Data":"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c"} Oct 30 17:36:21.153354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.152336 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerStarted","Data":"df5d3b9863ded677618ffb42fb2c9b8149289a4f14678093d9afdf7a78bdd92b"} Oct 30 17:36:21.278285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.249602 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podStartSLOduration=2.249581091 podStartE2EDuration="2.249581091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:21.222869558 +0000 UTC m=+1409.000341516" watchObservedRunningTime="2026-10-30 17:36:21.249581091 +0000 UTC m=+1409.027053046" Oct 30 17:36:21.863002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.859550 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:21.882199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.881189 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:36:21.909412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.907794 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:21.909412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.907901 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:21.909412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:21.907949 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:36:22.189674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.189566 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.218990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.218930 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.237574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.237535 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" event={"ID":"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db","Type":"ContainerDied","Data":"943a7ba83048a7af9026e8d2200339b9c4fae3acb6821ec4436ceda902e4c742"} Oct 30 17:36:22.237903 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.237879 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943a7ba83048a7af9026e8d2200339b9c4fae3acb6821ec4436ceda902e4c742" Oct 30 17:36:22.308285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.306451 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:22.308285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.306884 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.421286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.415370 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\" (UniqueName: \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\") pod \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " Oct 30 17:36:22.421286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.415596 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\") pod \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " Oct 30 17:36:22.421286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.415658 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\") pod \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\" (UID: \"4e2f86bf-fb41-4f6a-9a2c-76d10ad676db\") " Oct 30 17:36:22.422859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.422830 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca" pod "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" (UID: "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:22.480282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.478389 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.497308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.486506 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj" pod "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" (UID: "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db"). InnerVolumeSpecName "kube-api-access-b94tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:22.516425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.516386 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca" pod "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" (UID: "4e2f86bf-fb41-4f6a-9a2c-76d10ad676db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:22.554287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.530338 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:36:22.554287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.546393 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:22.578286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.555383 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94tj\" (UniqueName: \"kubernetes.io/projected/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-kube-api-access-b94tj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:36:22.578522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.578498 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:36:22.578657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.578642 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:36:22.882408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.882352 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:36:22.883037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.883015 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:36:22.899916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.899891 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:36:22.900587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.900561 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:36:22.993843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:22.993801 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.094567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.094537 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:36:23.217389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.195342 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:23.499526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.499427 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.499758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.350276 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 17:36:23.499888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.319510 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:23.637773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.635377 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:36:23.637773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.635444 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:36:23.637773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.637053 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf" Oct 30 17:36:23.982106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.981970 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:24.004327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:23.999501 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:24.135299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.133796 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:36:24.200365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.200331 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:24.225286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.200585 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:24.459159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.459127 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:36:24.461438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.461412 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:24.463486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:24.463455 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:36:24.583360 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:36:24.576105 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:36:43.898635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.898601 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 17:36:43.899694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.899666 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" containerName="revoke-keys" Oct 30 17:36:43.899694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.899691 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" containerName="revoke-keys" Oct 30 17:36:43.899905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.899864 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" containerName="revoke-keys" Oct 30 17:36:43.904369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.904344 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:43.932964 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.932934 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-8d1062-serving-cert\"" Oct 30 17:36:43.937105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.937076 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 17:36:43.946846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.946817 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:43.946964 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.946918 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpw8\" (UniqueName: \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:43.947243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.947219 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:43.947490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:43.947467 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.048682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.048646 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.048882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.048774 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.048882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.048801 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.048882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.048818 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpw8\" (UniqueName: \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.049496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.049473 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.051176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.051145 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.051365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.051343 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.073035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.073000 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpw8\" (UniqueName: \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\") pod \"maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.223235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.223151 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:44.558196 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:36:44.558157 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4987861_f6c8_4363_b616_14887a22b22d.slice/crio-4bc52d884310247be782e3b9844bf17441bb2c10c7d50bcc9b018ee56ff95293 WatchSource:0}: Error finding container 4bc52d884310247be782e3b9844bf17441bb2c10c7d50bcc9b018ee56ff95293: Status 404 returned error can't find the container with id 4bc52d884310247be782e3b9844bf17441bb2c10c7d50bcc9b018ee56ff95293 Oct 30 17:36:44.559105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.559084 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 17:36:44.656223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:44.656177 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" event={"ID":"c4987861-f6c8-4363-b616-14887a22b22d","Type":"ContainerStarted","Data":"4bc52d884310247be782e3b9844bf17441bb2c10c7d50bcc9b018ee56ff95293"} Oct 30 17:36:45.666516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:45.666476 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" event={"ID":"c4987861-f6c8-4363-b616-14887a22b22d","Type":"ContainerStarted","Data":"81f1bb46fe6773601233c9274db09f2a5ae1194a32e2471cd3809c87aee79ce2"} Oct 30 17:36:45.667059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:45.666550 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:45.713957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:45.713908 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" podStartSLOduration=2.270993854 podStartE2EDuration="2.713895907s" totalImagesPullingTime="442.902053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:45.709956761 +0000 UTC m=+1433.487428716" watchObservedRunningTime="2026-10-30 17:36:45.713895907 +0000 UTC m=+1433.491367862" Oct 30 17:36:51.679976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:51.679946 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 17:36:54.190494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:54.190448 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-v6484 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:54.191006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:36:54.190521 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:54.626429 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:36:54.626326 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:37:14.040521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:14.040487 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc"] Oct 30 17:37:14.054455 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:14.054419 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da079b02-wqlgc"] Oct 30 17:37:14.376527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:14.376449 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f69d903-2ca0-48bb-8065-b3108afba0e2" path="/var/lib/kubelet/pods/1f69d903-2ca0-48bb-8065-b3108afba0e2/volumes" Oct 30 17:37:16.043867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:16.043533 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb"] Oct 30 17:37:16.055293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:16.054178 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef32ce26-vnhpb"] Oct 30 17:37:16.377680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:16.377593 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3718ac1-af59-4f5e-9945-6127e0f7abea" path="/var/lib/kubelet/pods/a3718ac1-af59-4f5e-9945-6127e0f7abea/volumes" Oct 30 17:37:22.387808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.387723 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 17:37:22.402127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.402099 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.414277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.414238 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 17:37:22.417629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.417577 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-d53ec1-serving-cert\"" Oct 30 17:37:22.495651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.495619 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.495817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.495784 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.495979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.495958 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cvx\" (UniqueName: \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.496075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.495999 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.597519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.597481 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.597711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.597574 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.597711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.597647 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cvx\" (UniqueName: \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.597711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.597691 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.597898 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:22.597808 4622 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-serving-cert: secret "maas-api-e2e-shared-b-w0-d53ec1-serving-cert" not found Oct 30 17:37:22.597898 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:22.597886 4622 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls podName:67a256b9-aef1-4064-b94e-36957242d793 nodeName:}" failed. No retries permitted until 2026-10-30 17:37:23.097867722 +0000 UTC m=+1470.875339668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls") pod "maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" (UID: "67a256b9-aef1-4064-b94e-36957242d793") : secret "maas-api-e2e-shared-b-w0-d53ec1-serving-cert" not found Oct 30 17:37:22.598349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.598327 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.600309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.600283 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:22.624510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:22.624473 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cvx\" (UniqueName: \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:23.105950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:23.105909 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:23.112303 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:23.111094 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:23.331587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:23.331550 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:23.685254 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:23.685226 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a256b9_aef1_4064_b94e_36957242d793.slice/crio-dea00a83ed1df896461d627e0a7b7e629b79a4caa0ecad857aeca47ac27c6e9e WatchSource:0}: Error finding container dea00a83ed1df896461d627e0a7b7e629b79a4caa0ecad857aeca47ac27c6e9e: Status 404 returned error can't find the container with id dea00a83ed1df896461d627e0a7b7e629b79a4caa0ecad857aeca47ac27c6e9e Oct 30 17:37:23.686355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:23.686327 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 17:37:24.052730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:24.052692 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" event={"ID":"67a256b9-aef1-4064-b94e-36957242d793","Type":"ContainerStarted","Data":"dea00a83ed1df896461d627e0a7b7e629b79a4caa0ecad857aeca47ac27c6e9e"} Oct 30 17:37:24.538177 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:24.538132 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:37:25.063699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:25.063669 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" event={"ID":"67a256b9-aef1-4064-b94e-36957242d793","Type":"ContainerStarted","Data":"22404844d3863ebca50f1a1b51c7217cf6599ea0770cd9d3db83e579cb9b9a21"} Oct 30 17:37:25.064057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:25.063808 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:25.121761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:25.121718 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" podStartSLOduration=2.734481516 podStartE2EDuration="3.121706621s" totalImagesPullingTime="387.225105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:25.118856819 +0000 UTC m=+1472.896328774" watchObservedRunningTime="2026-10-30 17:37:25.121706621 +0000 UTC m=+1472.899178576" Oct 30 17:37:25.503464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:25.503383 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:37:25.503464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:25.503431 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:37:26.072234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.072164 4622 generic.go:356] "Generic (PLEG): container finished" podID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerID="248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c" exitCode=0 Oct 30 17:37:26.072629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.072232 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerDied","Data":"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c"} Oct 30 17:37:26.072694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.072680 4622 scope.go:122] "RemoveContainer" containerID="248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c" Oct 30 17:37:26.074179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.074156 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:26.074239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.074220 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:26.074289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:26.074252 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:27.081242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:27.081207 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerStarted","Data":"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417"} Oct 30 17:37:27.883607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:27.883577 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:37:27.883805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:27.883712 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:37:27.883805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:27.883768 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:37:28.884488 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:28.884455 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:37:28.884986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:28.884525 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:37:29.884110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:29.884065 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:37:29.884315 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:29.884127 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:37:30.890311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:30.886453 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:37:30.890311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:30.886515 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:37:31.078202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:31.078175 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 17:37:31.883773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:31.883737 4622 patch_prober.go:29] interesting pod/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:37:31.883932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:31.883799 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 17:37:32.888716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:32.888684 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:37:32.889193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:32.889172 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:37:32.889825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:32.889796 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:37:35.621863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.621827 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 17:37:35.640693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.640661 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.655822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.655792 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-4dcdbb-openshift-default-dockercfg-cq2mc\"" Oct 30 17:37:35.658535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.658508 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 17:37:35.734168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734131 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734181 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734366 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734445 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734468 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734552 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gbtt\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734621 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734661 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.734833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.734700 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.836695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836648 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gbtt\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.836888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836746 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.836888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836794 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.836888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836826 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.837060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836943 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.837060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.836994 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.837153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.837113 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.837225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.837211 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.837339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.837240 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.837843 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.839048 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.840719 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.841409 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.842622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.841714 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.844962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.844690 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.845632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.845586 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.882332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.873101 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.882332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.873773 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gbtt\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\") pod \"e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:35.975285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:35.974590 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:36.364001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.363971 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 17:37:36.364629 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:36.364603 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c400d2_e231_4269_bced_1a5235ae7cf2.slice/crio-76f5d3d77013de8e3bfa9c38c4843ead279b25bb0288540e059759c4f6541909 WatchSource:0}: Error finding container 76f5d3d77013de8e3bfa9c38c4843ead279b25bb0288540e059759c4f6541909: Status 404 returned error can't find the container with id 76f5d3d77013de8e3bfa9c38c4843ead279b25bb0288540e059759c4f6541909 Oct 30 17:37:36.368612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.368575 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:36.368736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.368659 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:36.368736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.368702 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:36.929132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.929097 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:37:36.934360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.934332 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.957600 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.957573 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:37:36.962503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.962479 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d1c973-openshift-default-dockercfg-tgrqz\"" Oct 30 17:37:36.985710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.985666 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bwh\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.985889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.985812 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.985889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.985848 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986007 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986145 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986304 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986347 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986423 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:36.986463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:36.986448 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117289 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117433 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117470 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117555 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117593 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117632 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bwh\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117702 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117731 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.117944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.117835 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.119328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.118432 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.135891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.122577 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.135891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.135580 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.135891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.135719 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.135891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.135844 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.161430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.161400 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bwh\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.172165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.172134 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.189330 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.174253 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.189330 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.188186 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\") pod \"e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.241294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.240030 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerStarted","Data":"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed"} Oct 30 17:37:37.241294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.240104 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerStarted","Data":"76f5d3d77013de8e3bfa9c38c4843ead279b25bb0288540e059759c4f6541909"} Oct 30 17:37:37.275287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.272840 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:37.663008 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.662922 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podStartSLOduration=2.662906744 podStartE2EDuration="2.662906744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:37.298346477 +0000 UTC m=+1485.075818433" watchObservedRunningTime="2026-10-30 17:37:37.662906744 +0000 UTC m=+1485.440378699" Oct 30 17:37:37.668293 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:37.667893 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadac9297_76e3_48da_aa15_3e66db6dec3f.slice/crio-3c16360ca2b67457e2be1ddaf1556d2777b171bfd0427414d7d4e27acbe57921 WatchSource:0}: Error finding container 3c16360ca2b67457e2be1ddaf1556d2777b171bfd0427414d7d4e27acbe57921: Status 404 returned error can't find the container with id 3c16360ca2b67457e2be1ddaf1556d2777b171bfd0427414d7d4e27acbe57921 Oct 30 17:37:37.671473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.669449 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:37:37.671756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.671721 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:37.671821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.671805 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:37.671871 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.671857 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:37:37.975113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.975025 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:37.980987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:37.980960 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:38.249101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.249013 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerStarted","Data":"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc"} Oct 30 17:37:38.249101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.249073 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerStarted","Data":"3c16360ca2b67457e2be1ddaf1556d2777b171bfd0427414d7d4e27acbe57921"} Oct 30 17:37:38.249336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.249204 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:38.250224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.250204 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:37:38.273297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.273276 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:38.279314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.279293 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:38.296154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:38.296116 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podStartSLOduration=2.2961027769999998 podStartE2EDuration="2.296102777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:38.29374216 +0000 UTC m=+1486.071214130" watchObservedRunningTime="2026-10-30 17:37:38.296102777 +0000 UTC m=+1486.073574731" Oct 30 17:37:39.256982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:39.256953 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:39.257913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:39.257897 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:37:44.100481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.100442 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8854-0f4d25"] Oct 30 17:37:44.107473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.107437 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:44.127344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.124467 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5c6gd\"" Oct 30 17:37:44.127344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.126400 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-0f4d25"] Oct 30 17:37:44.261365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.261333 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dk2p\" (UniqueName: \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\") pod \"test-curl-8854-0f4d25\" (UID: \"812e8155-3c12-447c-84e6-546e651a7a43\") " pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:44.362489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.362408 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dk2p\" (UniqueName: \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\") pod \"test-curl-8854-0f4d25\" (UID: \"812e8155-3c12-447c-84e6-546e651a7a43\") " pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:44.386563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.386530 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dk2p\" (UniqueName: \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\") pod \"test-curl-8854-0f4d25\" (UID: \"812e8155-3c12-447c-84e6-546e651a7a43\") " pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:44.471733 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.471702 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:44.793958 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:44.793931 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod812e8155_3c12_447c_84e6_546e651a7a43.slice/crio-eba0aae152b3073ac3c9303ce5202138e357d07d0479644ebb75a2cd81a84bd0 WatchSource:0}: Error finding container eba0aae152b3073ac3c9303ce5202138e357d07d0479644ebb75a2cd81a84bd0: Status 404 returned error can't find the container with id eba0aae152b3073ac3c9303ce5202138e357d07d0479644ebb75a2cd81a84bd0 Oct 30 17:37:44.795649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:44.795627 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-0f4d25"] Oct 30 17:37:45.306462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:45.306430 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-0f4d25" event={"ID":"812e8155-3c12-447c-84e6-546e651a7a43","Type":"ContainerStarted","Data":"faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030"} Oct 30 17:37:45.306462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:45.306473 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-0f4d25" event={"ID":"812e8155-3c12-447c-84e6-546e651a7a43","Type":"ContainerStarted","Data":"eba0aae152b3073ac3c9303ce5202138e357d07d0479644ebb75a2cd81a84bd0"} Oct 30 17:37:45.352677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:45.352631 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8854-0f4d25" podStartSLOduration=1.35261808 podStartE2EDuration="1.35261808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:45.349195146 +0000 UTC m=+1493.126667098" watchObservedRunningTime="2026-10-30 17:37:45.35261808 +0000 UTC m=+1493.130090035" Oct 30 17:37:46.293974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.293931 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8854-0f4d25"] Oct 30 17:37:46.301695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.301664 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8854-0f4d25"] Oct 30 17:37:46.461274 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.461220 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8854-b5cc3a"] Oct 30 17:37:46.461943 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.461926 4622 cpu_manager.go:414] "containerMap: removing container" podUID="812e8155-3c12-447c-84e6-546e651a7a43" containerName="test-curl-8854-0f4d25" Oct 30 17:37:46.461988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.461945 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="812e8155-3c12-447c-84e6-546e651a7a43" containerName="test-curl-8854-0f4d25" Oct 30 17:37:46.462080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.462071 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="812e8155-3c12-447c-84e6-546e651a7a43" containerName="test-curl-8854-0f4d25" Oct 30 17:37:46.465835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.465819 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:46.485955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.485919 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-b5cc3a"] Oct 30 17:37:46.493080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.493052 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4vch\" (UniqueName: \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\") pod \"test-curl-8854-b5cc3a\" (UID: \"d0bc3d69-665d-448a-9549-902b56402f30\") " pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:46.594415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.594325 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4vch\" (UniqueName: \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\") pod \"test-curl-8854-b5cc3a\" (UID: \"d0bc3d69-665d-448a-9549-902b56402f30\") " pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:46.618619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.618585 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4vch\" (UniqueName: \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\") pod \"test-curl-8854-b5cc3a\" (UID: \"d0bc3d69-665d-448a-9549-902b56402f30\") " pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:46.778662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:46.778626 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:47.090620 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:47.090578 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0bc3d69_665d_448a_9549_902b56402f30.slice/crio-bb308d8468b5e98698a54188d5b57fcbc26f3e0fdb86cb93cfa8ec9987343447 WatchSource:0}: Error finding container bb308d8468b5e98698a54188d5b57fcbc26f3e0fdb86cb93cfa8ec9987343447: Status 404 returned error can't find the container with id bb308d8468b5e98698a54188d5b57fcbc26f3e0fdb86cb93cfa8ec9987343447 Oct 30 17:37:47.092607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:47.092576 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-b5cc3a"] Oct 30 17:37:47.326336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:47.326304 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-b5cc3a" event={"ID":"d0bc3d69-665d-448a-9549-902b56402f30","Type":"ContainerStarted","Data":"69f713452d8e8a05ac83d6cd34d05f67796b8c67103593bc55c544b080a982df"} Oct 30 17:37:47.326540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:47.326353 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-b5cc3a" event={"ID":"d0bc3d69-665d-448a-9549-902b56402f30","Type":"ContainerStarted","Data":"bb308d8468b5e98698a54188d5b57fcbc26f3e0fdb86cb93cfa8ec9987343447"} Oct 30 17:37:47.326540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:47.326431 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8854-0f4d25" podUID="812e8155-3c12-447c-84e6-546e651a7a43" containerName="test-curl-8854-0f4d25" containerID="cri-o://faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030" gracePeriod=2 Oct 30 17:37:47.369194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:47.369096 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8854-b5cc3a" podStartSLOduration=1.369081983 podStartE2EDuration="1.369081983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:47.367594585 +0000 UTC m=+1495.145066539" watchObservedRunningTime="2026-10-30 17:37:47.369081983 +0000 UTC m=+1495.146553937" Oct 30 17:37:48.340562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.340529 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8854-b5cc3a"] Oct 30 17:37:48.346994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.346966 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8854-b5cc3a"] Oct 30 17:37:48.535067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.535033 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8854-4f7599"] Oct 30 17:37:48.535752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.535737 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d0bc3d69-665d-448a-9549-902b56402f30" containerName="test-curl-8854-b5cc3a" Oct 30 17:37:48.535805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.535754 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bc3d69-665d-448a-9549-902b56402f30" containerName="test-curl-8854-b5cc3a" Oct 30 17:37:48.535894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.535885 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bc3d69-665d-448a-9549-902b56402f30" containerName="test-curl-8854-b5cc3a" Oct 30 17:37:48.539611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.539596 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:37:48.579928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.579895 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-4f7599"] Oct 30 17:37:48.624951 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.624871 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sj5d\" (UniqueName: \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\") pod \"test-curl-8854-4f7599\" (UID: \"9e8abb3a-43aa-4796-88e4-f5a0e15250d9\") " pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:37:48.726163 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.726129 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sj5d\" (UniqueName: \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\") pod \"test-curl-8854-4f7599\" (UID: \"9e8abb3a-43aa-4796-88e4-f5a0e15250d9\") " pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:37:48.750969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.750935 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sj5d\" (UniqueName: \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\") pod \"test-curl-8854-4f7599\" (UID: \"9e8abb3a-43aa-4796-88e4-f5a0e15250d9\") " pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:37:48.853665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:48.853633 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:37:49.161819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.159999 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8854-4f7599"] Oct 30 17:37:49.163924 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:37:49.163896 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e8abb3a_43aa_4796_88e4_f5a0e15250d9.slice/crio-26586a5c7943f9890a147ba62ebb27f07654013a8f62f9ddc78e027833e1f6a5 WatchSource:0}: Error finding container 26586a5c7943f9890a147ba62ebb27f07654013a8f62f9ddc78e027833e1f6a5: Status 404 returned error can't find the container with id 26586a5c7943f9890a147ba62ebb27f07654013a8f62f9ddc78e027833e1f6a5 Oct 30 17:37:49.343947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.343918 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-4f7599" event={"ID":"9e8abb3a-43aa-4796-88e4-f5a0e15250d9","Type":"ContainerStarted","Data":"c480c9bd5063a7a5fdbf6f2f54e57e81e298410fdb9424b39803b98cea8af6ef"} Oct 30 17:37:49.344398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.343961 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8854-4f7599" event={"ID":"9e8abb3a-43aa-4796-88e4-f5a0e15250d9","Type":"ContainerStarted","Data":"26586a5c7943f9890a147ba62ebb27f07654013a8f62f9ddc78e027833e1f6a5"} Oct 30 17:37:49.344398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.344096 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8854-b5cc3a" podUID="d0bc3d69-665d-448a-9549-902b56402f30" containerName="test-curl-8854-b5cc3a" containerID="cri-o://69f713452d8e8a05ac83d6cd34d05f67796b8c67103593bc55c544b080a982df" gracePeriod=2 Oct 30 17:37:49.385095 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.385053 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8854-4f7599" podStartSLOduration=1.385039573 podStartE2EDuration="1.385039573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:49.384419276 +0000 UTC m=+1497.161891242" watchObservedRunningTime="2026-10-30 17:37:49.385039573 +0000 UTC m=+1497.162511527" Oct 30 17:37:49.523386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.523361 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:49.637816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.637771 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\" (UniqueName: \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\") pod \"812e8155-3c12-447c-84e6-546e651a7a43\" (UID: \"812e8155-3c12-447c-84e6-546e651a7a43\") " Oct 30 17:37:49.639982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.639947 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p" pod "812e8155-3c12-447c-84e6-546e651a7a43" (UID: "812e8155-3c12-447c-84e6-546e651a7a43"). InnerVolumeSpecName "kube-api-access-8dk2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:49.739714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:49.739625 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dk2p\" (UniqueName: \"kubernetes.io/projected/812e8155-3c12-447c-84e6-546e651a7a43-kube-api-access-8dk2p\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:37:50.354079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.354042 4622 generic.go:356] "Generic (PLEG): container finished" podID="812e8155-3c12-447c-84e6-546e651a7a43" containerID="faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030" exitCode=137 Oct 30 17:37:50.354539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.354087 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:50.354539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.354136 4622 scope.go:122] "RemoveContainer" containerID="faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030" Oct 30 17:37:50.364892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.364873 4622 scope.go:122] "RemoveContainer" containerID="faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030" Oct 30 17:37:50.365169 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:50.365151 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030\": container with ID starting with faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030 not found: ID does not exist" containerID="faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030" Oct 30 17:37:50.365217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.365178 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030"} err="failed to get container status \"faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030\": rpc error: code = NotFound desc = could not find container \"faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030\": container with ID starting with faf06fd167f1239cd6c28396c39ff46f196bf6b2fea03864ba9990aca5bcd030 not found: ID does not exist" Oct 30 17:37:50.375654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:50.375626 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="812e8155-3c12-447c-84e6-546e651a7a43" path="/var/lib/kubelet/pods/812e8155-3c12-447c-84e6-546e651a7a43/volumes" Oct 30 17:37:50.378876 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:50.378853 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-0f4d25\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="812e8155-3c12-447c-84e6-546e651a7a43" pod="opendatahub/test-curl-8854-0f4d25" Oct 30 17:37:51.370318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:51.370253 4622 generic.go:356] "Generic (PLEG): container finished" podID="d0bc3d69-665d-448a-9549-902b56402f30" containerID="69f713452d8e8a05ac83d6cd34d05f67796b8c67103593bc55c544b080a982df" exitCode=137 Oct 30 17:37:51.558619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:51.558591 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:51.678739 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:51.678661 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\" (UniqueName: \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\") pod \"d0bc3d69-665d-448a-9549-902b56402f30\" (UID: \"d0bc3d69-665d-448a-9549-902b56402f30\") " Oct 30 17:37:51.680752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:51.680719 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch" pod "d0bc3d69-665d-448a-9549-902b56402f30" (UID: "d0bc3d69-665d-448a-9549-902b56402f30"). InnerVolumeSpecName "kube-api-access-q4vch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:51.780499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:51.780469 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4vch\" (UniqueName: \"kubernetes.io/projected/d0bc3d69-665d-448a-9549-902b56402f30-kube-api-access-q4vch\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:37:52.385596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:52.385563 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bc3d69-665d-448a-9549-902b56402f30" path="/var/lib/kubelet/pods/d0bc3d69-665d-448a-9549-902b56402f30/volumes" Oct 30 17:37:52.388001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:52.387974 4622 scope.go:122] "RemoveContainer" containerID="69f713452d8e8a05ac83d6cd34d05f67796b8c67103593bc55c544b080a982df" Oct 30 17:37:52.388148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:37:52.387986 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-b5cc3a" Oct 30 17:37:54.713257 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:37:54.713202 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:38:02.570879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.570850 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8860-ee5fcd"] Oct 30 17:38:02.574924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.574909 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:02.592793 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.592762 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8860-ee5fcd"] Oct 30 17:38:02.681890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.681856 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6dsg\" (UniqueName: \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\") pod \"test-curl-8860-ee5fcd\" (UID: \"1b41134c-3238-4452-8c98-e715cc840064\") " pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:02.783368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.783331 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6dsg\" (UniqueName: \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\") pod \"test-curl-8860-ee5fcd\" (UID: \"1b41134c-3238-4452-8c98-e715cc840064\") " pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:02.807120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.807085 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6dsg\" (UniqueName: \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\") pod \"test-curl-8860-ee5fcd\" (UID: \"1b41134c-3238-4452-8c98-e715cc840064\") " pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:02.890233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:02.890140 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:03.192445 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:38:03.192414 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b41134c_3238_4452_8c98_e715cc840064.slice/crio-80431577110a9b1f66cde089fd387cfc4cbe4f3449839fae7909dd63020abd7a WatchSource:0}: Error finding container 80431577110a9b1f66cde089fd387cfc4cbe4f3449839fae7909dd63020abd7a: Status 404 returned error can't find the container with id 80431577110a9b1f66cde089fd387cfc4cbe4f3449839fae7909dd63020abd7a Oct 30 17:38:03.194492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:03.194398 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8860-ee5fcd"] Oct 30 17:38:03.485145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:03.485060 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8860-ee5fcd" event={"ID":"1b41134c-3238-4452-8c98-e715cc840064","Type":"ContainerStarted","Data":"d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707"} Oct 30 17:38:03.485145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:03.485099 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8860-ee5fcd" event={"ID":"1b41134c-3238-4452-8c98-e715cc840064","Type":"ContainerStarted","Data":"80431577110a9b1f66cde089fd387cfc4cbe4f3449839fae7909dd63020abd7a"} Oct 30 17:38:03.530339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:03.530296 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8860-ee5fcd" podStartSLOduration=1.530282405 podStartE2EDuration="1.530282405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:03.526459734 +0000 UTC m=+1511.303931688" watchObservedRunningTime="2026-10-30 17:38:03.530282405 +0000 UTC m=+1511.307754357" Oct 30 17:38:04.501024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:04.500978 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8860-ee5fcd"] Oct 30 17:38:04.509299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:04.507742 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8860-ee5fcd"] Oct 30 17:38:05.499465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:05.499429 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8860-ee5fcd" podUID="1b41134c-3238-4452-8c98-e715cc840064" containerName="test-curl-8860-ee5fcd" containerID="cri-o://d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707" gracePeriod=2 Oct 30 17:38:05.513234 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:38:05.513197 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8860-ee5fcd\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="1b41134c-3238-4452-8c98-e715cc840064" pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:07.723341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:07.723313 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:07.859486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:07.859455 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\" (UniqueName: \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\") pod \"1b41134c-3238-4452-8c98-e715cc840064\" (UID: \"1b41134c-3238-4452-8c98-e715cc840064\") " Oct 30 17:38:07.861513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:07.861481 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg" pod "1b41134c-3238-4452-8c98-e715cc840064" (UID: "1b41134c-3238-4452-8c98-e715cc840064"). InnerVolumeSpecName "kube-api-access-v6dsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:07.961917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:07.961878 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6dsg\" (UniqueName: \"kubernetes.io/projected/1b41134c-3238-4452-8c98-e715cc840064-kube-api-access-v6dsg\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:08.375388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.375357 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b41134c-3238-4452-8c98-e715cc840064" path="/var/lib/kubelet/pods/1b41134c-3238-4452-8c98-e715cc840064/volumes" Oct 30 17:38:08.522914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.522882 4622 generic.go:356] "Generic (PLEG): container finished" podID="1b41134c-3238-4452-8c98-e715cc840064" containerID="d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707" exitCode=137 Oct 30 17:38:08.523064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.522921 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-ee5fcd" Oct 30 17:38:08.523064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.522974 4622 scope.go:122] "RemoveContainer" containerID="d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707" Oct 30 17:38:08.533499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.533482 4622 scope.go:122] "RemoveContainer" containerID="d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707" Oct 30 17:38:08.533772 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:38:08.533749 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707\": container with ID starting with d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707 not found: ID does not exist" containerID="d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707" Oct 30 17:38:08.533868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:08.533778 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707"} err="failed to get container status \"d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707\": rpc error: code = NotFound desc = could not find container \"d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707\": container with ID starting with d02b4a068ddffdde5a9085ca54b471712cf327756d91815d7eea36dcdb7d8707 not found: ID does not exist" Oct 30 17:38:09.371853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:09.371822 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:38:09.372280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:09.371984 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" containerID="cri-o://053b6c01be206a7a2ec3ec20306b7a923d637a3fd9a4912cb24d80ac1a4797ec" gracePeriod=30 Oct 30 17:38:11.030599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:11.030565 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4"] Oct 30 17:38:11.042559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:11.042528 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54a5cb01-4xms4"] Oct 30 17:38:12.377030 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:12.376996 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6155c9ab-b99f-4508-be9b-b9583af577f4" path="/var/lib/kubelet/pods/6155c9ab-b99f-4508-be9b-b9583af577f4/volumes" Oct 30 17:38:14.592709 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.592669 4622 generic.go:356] "Generic (PLEG): container finished" podID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerID="053b6c01be206a7a2ec3ec20306b7a923d637a3fd9a4912cb24d80ac1a4797ec" exitCode=0 Oct 30 17:38:14.593108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.592740 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" event={"ID":"37d7627f-21fc-4cdf-b5d1-dcf817098b00","Type":"ContainerDied","Data":"053b6c01be206a7a2ec3ec20306b7a923d637a3fd9a4912cb24d80ac1a4797ec"} Oct 30 17:38:14.657438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.657381 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:38:14.761362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761324 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761393 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761423 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761472 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761524 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761619 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761694 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:14.761782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761712 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.761782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761714 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:14.761782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761762 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.762042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761879 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\") pod \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\" (UID: \"37d7627f-21fc-4cdf-b5d1-dcf817098b00\") " Oct 30 17:38:14.762042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761923 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:14.762042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.761933 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:14.762293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762213 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:14.762748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762725 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.762867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762754 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.762867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762770 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.762867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762785 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.762867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.762801 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.764117 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.764095 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "kube-api-access-6v94d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:14.764274 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.764236 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:14.764360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.764246 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:14.764732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.764699 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token" pod "37d7627f-21fc-4cdf-b5d1-dcf817098b00" (UID: "37d7627f-21fc-4cdf-b5d1-dcf817098b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:14.863995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.863959 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v94d\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-kube-api-access-6v94d\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.863995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.863991 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.863995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.864000 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:14.864301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:14.864009 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d7627f-21fc-4cdf-b5d1-dcf817098b00-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:38:15.602313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:15.602272 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" Oct 30 17:38:15.602313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:15.602289 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz" event={"ID":"37d7627f-21fc-4cdf-b5d1-dcf817098b00","Type":"ContainerDied","Data":"1cff6686889a857af7edb783ca32415fc1e129249b0b2203a186a01ab2cb902b"} Oct 30 17:38:15.602875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:15.602360 4622 scope.go:122] "RemoveContainer" containerID="053b6c01be206a7a2ec3ec20306b7a923d637a3fd9a4912cb24d80ac1a4797ec" Oct 30 17:38:15.679816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:15.679780 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:38:15.691886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:15.691854 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-vjwzz"] Oct 30 17:38:16.033646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:16.033618 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp"] Oct 30 17:38:16.037993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:16.037969 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2966cb1-n4ppp"] Oct 30 17:38:16.376906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:16.376811 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" path="/var/lib/kubelet/pods/37d7627f-21fc-4cdf-b5d1-dcf817098b00/volumes" Oct 30 17:38:16.377492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:16.377473 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff15c34-5f2d-4ca6-8834-d7ac8327ab52" path="/var/lib/kubelet/pods/aff15c34-5f2d-4ca6-8834-d7ac8327ab52/volumes" Oct 30 17:38:20.290033 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.289997 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 17:38:20.290683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290663 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1b41134c-3238-4452-8c98-e715cc840064" containerName="test-curl-8860-ee5fcd" Oct 30 17:38:20.290683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290682 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b41134c-3238-4452-8c98-e715cc840064" containerName="test-curl-8860-ee5fcd" Oct 30 17:38:20.290859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290725 4622 cpu_manager.go:414] "containerMap: removing container" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" Oct 30 17:38:20.290859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290731 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" Oct 30 17:38:20.290859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290830 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d7627f-21fc-4cdf-b5d1-dcf817098b00" containerName="istio-proxy" Oct 30 17:38:20.290859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.290840 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b41134c-3238-4452-8c98-e715cc840064" containerName="test-curl-8860-ee5fcd" Oct 30 17:38:20.296002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.295983 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.314829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.314801 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 17:38:20.324732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.324696 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-4dcdbb-serving-cert\"" Oct 30 17:38:20.455770 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.455731 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.455944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.455817 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvgg\" (UniqueName: \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.455993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.455978 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.456187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.456170 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.558099 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.558005 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.558294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.558152 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.558389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.558324 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.558389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.558375 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvgg\" (UniqueName: \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.558804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.558780 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.560626 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.560604 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.560714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.560673 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.586586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.586560 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvgg\" (UniqueName: \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\") pod \"maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.610784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.610752 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:20.930668 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:38:20.930628 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150533ae_1f35_493b_bc3c_525065322660.slice/crio-7bbc1e366bf3fc4e0115df20921e76ae5e220a1e074c602f3a5f393dcf9f2534 WatchSource:0}: Error finding container 7bbc1e366bf3fc4e0115df20921e76ae5e220a1e074c602f3a5f393dcf9f2534: Status 404 returned error can't find the container with id 7bbc1e366bf3fc4e0115df20921e76ae5e220a1e074c602f3a5f393dcf9f2534 Oct 30 17:38:20.931538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:20.931513 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 17:38:21.657103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:21.657061 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" event={"ID":"150533ae-1f35-493b-bc3c-525065322660","Type":"ContainerStarted","Data":"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94"} Oct 30 17:38:21.657572 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:21.657155 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:21.657572 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:21.657174 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" event={"ID":"150533ae-1f35-493b-bc3c-525065322660","Type":"ContainerStarted","Data":"7bbc1e366bf3fc4e0115df20921e76ae5e220a1e074c602f3a5f393dcf9f2534"} Oct 30 17:38:21.701643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:21.701554 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" podStartSLOduration=1.191193666 podStartE2EDuration="1.701539682s" totalImagesPullingTime="510.346016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:21.699505382 +0000 UTC m=+1529.476977337" watchObservedRunningTime="2026-10-30 17:38:21.701539682 +0000 UTC m=+1529.479011636" Oct 30 17:38:24.547909 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:38:24.547870 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:38:27.670390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:27.670360 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 17:38:53.873434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:53.873360 4622 scope.go:122] "RemoveContainer" containerID="b37c03fe132f7940ed507caa6df5cf0d3e6332e94fd05cadd30465fcd665c344" Oct 30 17:38:53.884686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:53.884667 4622 scope.go:122] "RemoveContainer" containerID="4c7e6bf02651d4460e0ed945cdd6071d889f0905f8cf47f16c5c11b8a21b7144" Oct 30 17:38:54.538921 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:38:54.538869 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:38:59.286352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:59.285314 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8854-4f7599"] Oct 30 17:38:59.286352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:59.285561 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8854-4f7599" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" containerName="test-curl-8854-4f7599" containerID="cri-o://c480c9bd5063a7a5fdbf6f2f54e57e81e298410fdb9424b39803b98cea8af6ef" gracePeriod=2 Oct 30 17:38:59.291832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:38:59.291802 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8854-4f7599"] Oct 30 17:38:59.299701 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:38:59.299658 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:02.226359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.226248 4622 generic.go:356] "Generic (PLEG): container finished" podID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" containerID="c480c9bd5063a7a5fdbf6f2f54e57e81e298410fdb9424b39803b98cea8af6ef" exitCode=137 Oct 30 17:39:02.529106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.529065 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:02.686334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.686299 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:02.717557 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:02.717514 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:02.860586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.860542 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\" (UniqueName: \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\") pod \"9e8abb3a-43aa-4796-88e4-f5a0e15250d9\" (UID: \"9e8abb3a-43aa-4796-88e4-f5a0e15250d9\") " Oct 30 17:39:02.876474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.876434 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d" pod "9e8abb3a-43aa-4796-88e4-f5a0e15250d9" (UID: "9e8abb3a-43aa-4796-88e4-f5a0e15250d9"). InnerVolumeSpecName "kube-api-access-7sj5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:02.974530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:02.974484 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sj5d\" (UniqueName: \"kubernetes.io/projected/9e8abb3a-43aa-4796-88e4-f5a0e15250d9-kube-api-access-7sj5d\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:03.204435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.204337 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:03.222422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.222347 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:03.247140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.247114 4622 scope.go:122] "RemoveContainer" containerID="c480c9bd5063a7a5fdbf6f2f54e57e81e298410fdb9424b39803b98cea8af6ef" Oct 30 17:39:03.251463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.251440 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:03.284128 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:03.283969 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:03.298848 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:03.298789 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:03.675288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.664876 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:39:03.675288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.672885 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:39:03.758298 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:03.740140 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:03.866318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.844030 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:03.895348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:03.895284 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.072562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.072524 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:39:04.132230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.124450 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.162639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.156179 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.219818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.217572 4622 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.237305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.228342 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:04.237305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.228731 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:39:04.238925 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:04.238885 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:04.435691 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:04.425994 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:04.960212 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:04.960165 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:39:04.984241 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:04.984198 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8854-4f7599\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" pod="opendatahub/test-curl-8854-4f7599" Oct 30 17:39:05.440776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:05.440737 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" path="/var/lib/kubelet/pods/9e8abb3a-43aa-4796-88e4-f5a0e15250d9/volumes" Oct 30 17:39:05.441875 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:05.441850 4622 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.067s" Oct 30 17:39:05.442054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:05.442041 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 17:39:05.442234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:05.442222 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:39:05.968395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:05.968348 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Oct 30 17:39:06.085448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.085398 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.334880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.334822 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:39:06.337124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.335757 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.369294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.335794 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.391767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.390551 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.493443 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.493398 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.787182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.787142 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:39:06.787479 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.787450 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:06.978466 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:06.978416 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:07.137159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.135131 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:39:07.137159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.135881 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:39:07.386627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.384563 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:39:07.422342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.389250 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.97:8000/ready\": context deadline exceeded" Oct 30 17:39:07.422763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.390549 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:39:07.422987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.422972 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 17:39:07.423131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.423119 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 17:39:07.426323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.408351 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 17:39:07.467829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.427121 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:39:07.468144 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.468122 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:39:07.470220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.470188 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:07.607653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.607610 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:07.608875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.608849 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:39:07.634663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.630752 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 17:39:07.701809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.688765 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 17:39:07.701809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.692331 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 17:39:07.739280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:07.725797 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 17:39:09.296140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:09.295753 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:39:09.296140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:09.295817 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:10.300046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:10.299508 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:39:10.300046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:10.299572 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:14.035537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.035500 4622 generic.go:356] "Generic (PLEG): container finished" podID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerID="018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed" exitCode=0 Oct 30 17:39:14.036108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.035570 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerDied","Data":"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed"} Oct 30 17:39:14.037892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.037869 4622 scope.go:122] "RemoveContainer" containerID="018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed" Oct 30 17:39:14.039369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.039340 4622 generic.go:356] "Generic (PLEG): container finished" podID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerID="88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc" exitCode=0 Oct 30 17:39:14.039988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.039657 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerDied","Data":"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc"} Oct 30 17:39:14.040803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.040668 4622 scope.go:122] "RemoveContainer" containerID="88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc" Oct 30 17:39:14.040803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.040710 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:14.040803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.040809 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:14.041100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.040856 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:14.043950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.043915 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:14.044049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.044013 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:14.044126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:14.044065 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:39:15.114127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:15.113589 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerStarted","Data":"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2"} Oct 30 17:39:15.116808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:15.116783 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerStarted","Data":"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7"} Oct 30 17:39:15.310828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:15.310789 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:39:15.311019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:15.310906 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:15.311019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:15.310944 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:16.008769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.000436 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:39:16.008769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.001304 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 17:39:16.008769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.001360 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 17:39:16.317653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.317621 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:16.318120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.317678 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:16.988534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.987521 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 17:39:16.988534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:16.987622 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 17:39:17.289286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:17.285365 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:17.289286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:17.285439 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:17.990896 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:17.990850 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 17:39:17.991410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:17.990925 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 17:39:18.291655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:18.291623 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:18.292453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:18.292423 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:18.977288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:18.976316 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 17:39:18.977288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:18.976382 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 17:39:19.288565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:19.282532 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:19.288565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:19.282590 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:19.986085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:19.985630 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 17:39:19.986085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:19.985694 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 17:39:20.283770 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:20.283736 4622 patch_prober.go:29] interesting pod/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:39:20.284054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:20.284024 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 17:39:20.979565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:20.979535 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:39:20.980058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:20.980046 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:39:20.980706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:20.980685 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 17:39:21.278519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:21.278444 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:39:21.278762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:21.278738 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:39:21.279323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:21.279304 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:39:24.535549 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:24.535507 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:39:36.789504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:36.789444 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:39:36.790737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:36.790700 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:39:39.560982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.560949 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:39:39.561436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.561134 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" containerID="cri-o://3d8363a7a37873370212616a937234f9182a4cf1d7f39da4172146507da51e87" gracePeriod=30 Oct 30 17:39:39.647178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.647139 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:39:39.647435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.647409 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" containerID="cri-o://11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568" gracePeriod=30 Oct 30 17:39:39.652459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.652430 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:39:39.652685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.652660 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" containerID="cri-o://cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83" gracePeriod=30 Oct 30 17:39:39.784283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.784235 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:39:39.784550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.784519 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" containerID="cri-o://3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec" gracePeriod=30 Oct 30 17:39:39.821040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.820953 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:39:39.821225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:39.821197 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" containerID="cri-o://ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435" gracePeriod=30 Oct 30 17:39:44.749562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:44.749531 4622 generic.go:356] "Generic (PLEG): container finished" podID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerID="3d8363a7a37873370212616a937234f9182a4cf1d7f39da4172146507da51e87" exitCode=0 Oct 30 17:39:44.749898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:44.749604 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" event={"ID":"7f21d5fe-e150-4cd3-b648-25a0870d4bf9","Type":"ContainerDied","Data":"3d8363a7a37873370212616a937234f9182a4cf1d7f39da4172146507da51e87"} Oct 30 17:39:44.965226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:44.965193 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:39:45.012412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.012378 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.073549 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.073595 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.073653 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.073735 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.073775 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074011 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074057 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074117 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074236 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074293 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074356 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074395 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\") pod \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\" (UID: \"7f21d5fe-e150-4cd3-b648-25a0870d4bf9\") " Oct 30 17:39:45.074445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074420 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074587 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.074801 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.075023 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.075020 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.075129 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.075146 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.075629 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.075553 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076400 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076426 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076440 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076453 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076466 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076480 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076495 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.076530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.076507 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.079478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.079415 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.079585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.079557 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:45.081171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.081135 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.086313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.085038 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8" pod "7f21d5fe-e150-4cd3-b648-25a0870d4bf9" (UID: "7f21d5fe-e150-4cd3-b648-25a0870d4bf9"). InnerVolumeSpecName "kube-api-access-89bx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.089912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.089614 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:39:45.153409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.153366 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:39:45.177447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177418 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177452 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177527 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177556 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177581 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177605 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177630 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.177662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177654 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177671 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177707 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177736 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177758 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177808 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177837 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177867 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177895 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177921 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.178052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.177977 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\") pod \"623ee1b6-3540-4022-8b7f-ab0696b1423b\" (UID: \"623ee1b6-3540-4022-8b7f-ab0696b1423b\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178088 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178125 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178288 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178379 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178413 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\") pod \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\" (UID: \"f7ca6351-9809-4dde-8040-fe18ad9e41fa\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178474 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178511 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\") pod \"df8523b8-4fec-423e-902b-9f01c2788e1d\" (UID: \"df8523b8-4fec-423e-902b-9f01c2788e1d\") " Oct 30 17:39:45.178943 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178739 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:45.179020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.178998 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.179020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179011 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.179458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179433 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179462 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179475 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179487 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179500 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179513 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179527 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bx8\" (UniqueName: \"kubernetes.io/projected/7f21d5fe-e150-4cd3-b648-25a0870d4bf9-kube-api-access-89bx8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179539 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.179945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179748 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.179945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.179918 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.180119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.180098 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.180811 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.180728 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.180924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.180828 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.181512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.181135 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:45.181512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.181183 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.181893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.181854 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:45.183451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.183415 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.183553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.183501 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "kube-api-access-lhpqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.184560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.184536 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:45.184659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.184572 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.184718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.184697 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.184768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.184732 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.185092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.185069 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.186618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.186582 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.186703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.186687 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "kube-api-access-7x9xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.186761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.186720 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8" pod "f7ca6351-9809-4dde-8040-fe18ad9e41fa" (UID: "f7ca6351-9809-4dde-8040-fe18ad9e41fa"). InnerVolumeSpecName "kube-api-access-d26v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.187118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.186758 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo" pod "df8523b8-4fec-423e-902b-9f01c2788e1d" (UID: "df8523b8-4fec-423e-902b-9f01c2788e1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:45.187118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.187025 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo" pod "623ee1b6-3540-4022-8b7f-ab0696b1423b" (UID: "623ee1b6-3540-4022-8b7f-ab0696b1423b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:45.281353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281317 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281349 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281361 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d26v8\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-kube-api-access-d26v8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281369 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8523b8-4fec-423e-902b-9f01c2788e1d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281378 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhpqr\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-kube-api-access-lhpqr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281387 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281395 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281403 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9xt\" (UniqueName: \"kubernetes.io/projected/623ee1b6-3540-4022-8b7f-ab0696b1423b-kube-api-access-7x9xt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281411 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281419 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281426 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281433 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7ca6351-9809-4dde-8040-fe18ad9e41fa-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281441 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281450 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281458 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623ee1b6-3540-4022-8b7f-ab0696b1423b-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281473 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281481 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281489 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8523b8-4fec-423e-902b-9f01c2788e1d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281496 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.281609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.281505 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623ee1b6-3540-4022-8b7f-ab0696b1423b-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.451401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.451373 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:39:45.483901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.483866 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.483928 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484105 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484180 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.484239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484222 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484250 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484315 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484352 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484410 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484456 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.484540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484381 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:45.484540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484526 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\") pod \"2c0c7c85-8afb-483e-aaba-b37da00849c5\" (UID: \"2c0c7c85-8afb-483e-aaba-b37da00849c5\") " Oct 30 17:39:45.484742 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484559 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.484742 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.484697 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.485126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.485094 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.485126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.485116 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0c7c85-8afb-483e-aaba-b37da00849c5-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.485126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.485128 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.485386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.485140 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.485386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.485154 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.486436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.486417 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:45.486731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.486712 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.486876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.486856 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:45.486963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.486878 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl" pod "2c0c7c85-8afb-483e-aaba-b37da00849c5" (UID: "2c0c7c85-8afb-483e-aaba-b37da00849c5"). InnerVolumeSpecName "kube-api-access-mxfhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:45.586078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.586048 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.586078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.586075 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.586078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.586086 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0c7c85-8afb-483e-aaba-b37da00849c5-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.586355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.586095 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfhl\" (UniqueName: \"kubernetes.io/projected/2c0c7c85-8afb-483e-aaba-b37da00849c5-kube-api-access-mxfhl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:45.766338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.761567 4622 generic.go:356] "Generic (PLEG): container finished" podID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerID="ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435" exitCode=0 Oct 30 17:39:45.766338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.761663 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" event={"ID":"df8523b8-4fec-423e-902b-9f01c2788e1d","Type":"ContainerDied","Data":"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435"} Oct 30 17:39:45.766338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.761714 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" event={"ID":"df8523b8-4fec-423e-902b-9f01c2788e1d","Type":"ContainerDied","Data":"ac693d7d11ddb412704200ce1cdf26854f8a81717f331984ad136f17b2f886b5"} Oct 30 17:39:45.766338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.761749 4622 scope.go:122] "RemoveContainer" containerID="ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435" Oct 30 17:39:45.766338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.761910 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf" Oct 30 17:39:45.772488 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.772454 4622 generic.go:356] "Generic (PLEG): container finished" podID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerID="11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568" exitCode=0 Oct 30 17:39:45.772916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.772744 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" Oct 30 17:39:45.776602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.772765 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" event={"ID":"623ee1b6-3540-4022-8b7f-ab0696b1423b","Type":"ContainerDied","Data":"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568"} Oct 30 17:39:45.776868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.776812 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l" event={"ID":"623ee1b6-3540-4022-8b7f-ab0696b1423b","Type":"ContainerDied","Data":"8c98cd0547a19aced8a9a5ff7a775149cd38be612a789b8dad67f9f23766c8f5"} Oct 30 17:39:45.778439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.777552 4622 generic.go:356] "Generic (PLEG): container finished" podID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerID="cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83" exitCode=0 Oct 30 17:39:45.778439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.777645 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" event={"ID":"f7ca6351-9809-4dde-8040-fe18ad9e41fa","Type":"ContainerDied","Data":"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83"} Oct 30 17:39:45.778439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.777682 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" event={"ID":"f7ca6351-9809-4dde-8040-fe18ad9e41fa","Type":"ContainerDied","Data":"daec2ea5ec22b5327411b46b0b16d9093b9dc7fed28f8841a6ccb06a5c58e16b"} Oct 30 17:39:45.778439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.777766 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6" Oct 30 17:39:45.781290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.781237 4622 generic.go:356] "Generic (PLEG): container finished" podID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerID="3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec" exitCode=0 Oct 30 17:39:45.781396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.781362 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" Oct 30 17:39:45.781480 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.781453 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" event={"ID":"2c0c7c85-8afb-483e-aaba-b37da00849c5","Type":"ContainerDied","Data":"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec"} Oct 30 17:39:45.781533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.781512 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb" event={"ID":"2c0c7c85-8afb-483e-aaba-b37da00849c5","Type":"ContainerDied","Data":"f844e1521b1eee8b778225b89f934ea017b625367e1fa417579f3665e8a076cf"} Oct 30 17:39:45.789644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.784403 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" event={"ID":"7f21d5fe-e150-4cd3-b648-25a0870d4bf9","Type":"ContainerDied","Data":"8c8da560c330a98155c5a0d6724ca351625945478ca8eee655fd006c4f019208"} Oct 30 17:39:45.789644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.784518 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss" Oct 30 17:39:45.795819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.795795 4622 scope.go:122] "RemoveContainer" containerID="ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435" Oct 30 17:39:45.796611 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:45.796583 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435\": container with ID starting with ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435 not found: ID does not exist" containerID="ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435" Oct 30 17:39:45.796722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.796622 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435"} err="failed to get container status \"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435\": rpc error: code = NotFound desc = could not find container \"ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435\": container with ID starting with ff5becc575622730c5dc7e3882682d2e1e2bdb38f737f925b4f626c40a416435 not found: ID does not exist" Oct 30 17:39:45.796722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.796650 4622 scope.go:122] "RemoveContainer" containerID="11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568" Oct 30 17:39:45.809911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.809893 4622 scope.go:122] "RemoveContainer" containerID="11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568" Oct 30 17:39:45.810238 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:45.810213 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568\": container with ID starting with 11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568 not found: ID does not exist" containerID="11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568" Oct 30 17:39:45.810358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.810246 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568"} err="failed to get container status \"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568\": rpc error: code = NotFound desc = could not find container \"11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568\": container with ID starting with 11475e395e7e4bf2e6e1877973e982b2c8f9c248534266eac968e8a59bc8b568 not found: ID does not exist" Oct 30 17:39:45.810358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.810275 4622 scope.go:122] "RemoveContainer" containerID="cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83" Oct 30 17:39:45.820992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.820969 4622 scope.go:122] "RemoveContainer" containerID="cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83" Oct 30 17:39:45.821497 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:45.821469 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83\": container with ID starting with cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83 not found: ID does not exist" containerID="cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83" Oct 30 17:39:45.821599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.821523 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83"} err="failed to get container status \"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83\": rpc error: code = NotFound desc = could not find container \"cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83\": container with ID starting with cd1f517e437b75094e2391ee982e255190c8a5dcbc2bdff974b2153959b95a83 not found: ID does not exist" Oct 30 17:39:45.821599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.821550 4622 scope.go:122] "RemoveContainer" containerID="3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec" Oct 30 17:39:45.833879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.833856 4622 scope.go:122] "RemoveContainer" containerID="3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec" Oct 30 17:39:45.834225 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:45.834206 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec\": container with ID starting with 3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec not found: ID does not exist" containerID="3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec" Oct 30 17:39:45.834314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.834231 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec"} err="failed to get container status \"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec\": rpc error: code = NotFound desc = could not find container \"3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec\": container with ID starting with 3077cddcd505be8e5a72ea71fac7e679840bb52de228e4ff0f3006a8dba352ec not found: ID does not exist" Oct 30 17:39:45.834314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.834252 4622 scope.go:122] "RemoveContainer" containerID="3d8363a7a37873370212616a937234f9182a4cf1d7f39da4172146507da51e87" Oct 30 17:39:45.856722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.856690 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:39:45.860886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.860860 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-f4c7l"] Oct 30 17:39:45.895606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.895374 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:39:45.899907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.899881 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-pcqss"] Oct 30 17:39:45.939180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.939150 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:39:45.943071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.943048 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-lg8t6"] Oct 30 17:39:45.976164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.976129 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:39:45.984789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:45.984752 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-c8nsf"] Oct 30 17:39:46.014003 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.013969 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:39:46.018765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.018699 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-gj8wb"] Oct 30 17:39:46.375000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.374960 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" path="/var/lib/kubelet/pods/2c0c7c85-8afb-483e-aaba-b37da00849c5/volumes" Oct 30 17:39:46.375636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.375614 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" path="/var/lib/kubelet/pods/623ee1b6-3540-4022-8b7f-ab0696b1423b/volumes" Oct 30 17:39:46.376179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.376160 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" path="/var/lib/kubelet/pods/7f21d5fe-e150-4cd3-b648-25a0870d4bf9/volumes" Oct 30 17:39:46.377005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.376985 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" path="/var/lib/kubelet/pods/df8523b8-4fec-423e-902b-9f01c2788e1d/volumes" Oct 30 17:39:46.377503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:46.377485 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" path="/var/lib/kubelet/pods/f7ca6351-9809-4dde-8040-fe18ad9e41fa/volumes" Oct 30 17:39:50.978831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:50.977848 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:39:50.978831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:50.978125 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" containerID="cri-o://9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417" gracePeriod=30 Oct 30 17:39:53.927575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:53.927543 4622 scope.go:122] "RemoveContainer" containerID="156d9c918cb368882b4449c97052951c683ec99b738c2fe2f7e1298797677ed5" Oct 30 17:39:53.939766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:53.939743 4622 scope.go:122] "RemoveContainer" containerID="dc700b3c1843ea0c32db66896fc305b99a5572cc14181303a71f4acd8db7b7de" Oct 30 17:39:54.534781 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:54.534748 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:39:56.272700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.272673 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:39:56.360681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.360651 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.360858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.360730 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.360858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.360798 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.360991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.360886 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.360991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.360977 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.361106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361013 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.361106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361018 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:56.361106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361061 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.361250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361139 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.361250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361217 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:56.361250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361224 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\") pod \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\" (UID: \"b809348b-21fb-4bcd-a068-c9b06a6f0d40\") " Oct 30 17:39:56.361458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361423 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361677 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.361984 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.362006 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.362021 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.362035 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.362233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.362195 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:56.363718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.363655 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:56.363832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.363749 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:56.363895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.363851 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:56.364149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.364126 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l" pod "b809348b-21fb-4bcd-a068-c9b06a6f0d40" (UID: "b809348b-21fb-4bcd-a068-c9b06a6f0d40"). InnerVolumeSpecName "kube-api-access-nmg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:56.463527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.463504 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.463527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.463527 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmg6l\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-kube-api-access-nmg6l\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.463693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.463539 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.463693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.463565 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.463693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.463591 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b809348b-21fb-4bcd-a068-c9b06a6f0d40-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:39:56.903116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.903078 4622 generic.go:356] "Generic (PLEG): container finished" podID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerID="9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417" exitCode=0 Oct 30 17:39:56.903343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.903136 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" Oct 30 17:39:56.903343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.903160 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerDied","Data":"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417"} Oct 30 17:39:56.903343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.903225 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh" event={"ID":"b809348b-21fb-4bcd-a068-c9b06a6f0d40","Type":"ContainerDied","Data":"df5d3b9863ded677618ffb42fb2c9b8149289a4f14678093d9afdf7a78bdd92b"} Oct 30 17:39:56.903343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.903256 4622 scope.go:122] "RemoveContainer" containerID="9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417" Oct 30 17:39:56.914084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.914066 4622 scope.go:122] "RemoveContainer" containerID="248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c" Oct 30 17:39:56.923523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.923503 4622 scope.go:122] "RemoveContainer" containerID="9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417" Oct 30 17:39:56.923804 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:56.923782 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417\": container with ID starting with 9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417 not found: ID does not exist" containerID="9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417" Oct 30 17:39:56.923852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.923815 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417"} err="failed to get container status \"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417\": rpc error: code = NotFound desc = could not find container \"9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417\": container with ID starting with 9403848792a73c6a62e768af51e0521c2bf0c25fe856b533811ac0b15fe35417 not found: ID does not exist" Oct 30 17:39:56.923852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.923832 4622 scope.go:122] "RemoveContainer" containerID="248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c" Oct 30 17:39:56.924061 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:39:56.924045 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c\": container with ID starting with 248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c not found: ID does not exist" containerID="248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c" Oct 30 17:39:56.924113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.924064 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c"} err="failed to get container status \"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c\": rpc error: code = NotFound desc = could not find container \"248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c\": container with ID starting with 248d9bc0035a91560e0168a3dd0f2cc851ee3d1c3ebb151d6a6c65985729c94c not found: ID does not exist" Oct 30 17:39:56.974140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.974102 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:39:56.988131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:56.988098 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df8b20-openshift-default-6598475757-9xfbh"] Oct 30 17:39:58.375681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:39:58.375654 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" path="/var/lib/kubelet/pods/b809348b-21fb-4bcd-a068-c9b06a6f0d40/volumes" Oct 30 17:40:16.611548 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.611510 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:40:16.612474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612451 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612475 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612526 4622 cpu_manager.go:414] "containerMap: removing container" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612534 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612549 4622 cpu_manager.go:414] "containerMap: removing container" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" containerName="test-curl-8854-4f7599" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612557 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" containerName="test-curl-8854-4f7599" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612576 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612583 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" Oct 30 17:40:16.612601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612603 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612610 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612624 4622 cpu_manager.go:414] "containerMap: removing container" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612631 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612648 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612655 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612804 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612817 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0c7c85-8afb-483e-aaba-b37da00849c5" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612829 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="623ee1b6-3540-4022-8b7f-ab0696b1423b" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612840 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ca6351-9809-4dde-8040-fe18ad9e41fa" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612859 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8523b8-4fec-423e-902b-9f01c2788e1d" containerName="istio-proxy" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612868 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8abb3a-43aa-4796-88e4-f5a0e15250d9" containerName="test-curl-8854-4f7599" Oct 30 17:40:16.613024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.612879 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f21d5fe-e150-4cd3-b648-25a0870d4bf9" containerName="istio-proxy" Oct 30 17:40:16.617588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.617568 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.640727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.640691 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:40:16.649617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.649565 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-49e155-openshift-default-dockercfg-gzfdq\"" Oct 30 17:40:16.675746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675720 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.675898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675776 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.675898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675810 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.675898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675877 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.676057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675905 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.676057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.675983 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.676057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.676047 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.676231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.676132 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.676420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.676323 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tlzr\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.777763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.777721 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.777963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.777836 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tlzr\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.777963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.777923 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.777964 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.777994 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778053 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778080 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778152 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.778354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778209 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.779054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778472 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.779054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778626 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.779054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.778689 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.779054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.779003 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.779633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.779607 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.780738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.780712 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.802301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.792501 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.826293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.820363 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.837046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.836953 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tlzr\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\") pod \"e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:16.954957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:16.953611 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:17.342594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:17.342543 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:40:17.351910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:17.351873 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:17.352039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:17.351962 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:17.352039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:17.352011 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:18.082305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:18.082248 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerStarted","Data":"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4"} Oct 30 17:40:18.082803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:18.082367 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerStarted","Data":"c3278bf8aafa7d9a2a0bf78a76704faef956e33b07be44efbc460f5b2e1354c1"} Oct 30 17:40:18.134678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:18.134625 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podStartSLOduration=2.1346057959999998 podStartE2EDuration="2.134605796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:18.130938025 +0000 UTC m=+1645.908409980" watchObservedRunningTime="2026-10-30 17:40:18.134605796 +0000 UTC m=+1645.912077753" Oct 30 17:40:18.954147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:18.954112 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:18.960976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:18.960926 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:19.090405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:19.090366 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:19.091344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:19.091326 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:40:24.535605 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:24.535522 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:40:24.936400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.936306 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs"] Oct 30 17:40:24.937528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.937501 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:24.937528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.937526 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:24.937748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.937685 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b809348b-21fb-4bcd-a068-c9b06a6f0d40" containerName="istio-proxy" Oct 30 17:40:24.941954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.941933 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:24.964305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.964252 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts2f\" (UniqueName: \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:24.964500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.964480 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:24.964684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.964668 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:24.965824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.965799 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:40:24.965944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.965870 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:40:24.967457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:24.967434 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs"] Oct 30 17:40:25.067810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.067775 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.068040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.067905 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts2f\" (UniqueName: \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.068040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.068024 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.068600 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.068574 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.070911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.070882 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.092979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.092948 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts2f\" (UniqueName: \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\") pod \"maas-api-revoke-sub-fc4aa170-vmvjs\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.232416 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.232329 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:40:25.232582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.232523 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" containerID="cri-o://f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255" gracePeriod=30 Oct 30 17:40:25.236511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.236485 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:40:25.236700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.236660 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" containerID="cri-o://f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e" gracePeriod=30 Oct 30 17:40:25.243932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.243907 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:40:25.244135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.244107 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" containerID="cri-o://73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d" gracePeriod=30 Oct 30 17:40:25.267997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.267960 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:25.585041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:25.585010 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs"] Oct 30 17:40:25.585470 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:25.585427 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7466eeae_c194_42ea_ac6a_1fb3e8aa675a.slice/crio-0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384 WatchSource:0}: Error finding container 0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384: Status 404 returned error can't find the container with id 0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384 Oct 30 17:40:26.146663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:26.146586 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" event={"ID":"7466eeae-c194-42ea-ac6a-1fb3e8aa675a","Type":"ContainerStarted","Data":"039972031a92faba3621d3699e668c2bc6059ee6aeddc3948cae13fe7188b2c4"} Oct 30 17:40:26.146663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:26.146635 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" event={"ID":"7466eeae-c194-42ea-ac6a-1fb3e8aa675a","Type":"ContainerStarted","Data":"0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384"} Oct 30 17:40:26.188189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:26.188140 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" podStartSLOduration=2.188127379 podStartE2EDuration="2.188127379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:26.187151891 +0000 UTC m=+1653.964623842" watchObservedRunningTime="2026-10-30 17:40:26.188127379 +0000 UTC m=+1653.965599335" Oct 30 17:40:27.154914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:27.154881 4622 generic.go:356] "Generic (PLEG): container finished" podID="7466eeae-c194-42ea-ac6a-1fb3e8aa675a" containerID="039972031a92faba3621d3699e668c2bc6059ee6aeddc3948cae13fe7188b2c4" exitCode=0 Oct 30 17:40:27.155371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:27.154963 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" event={"ID":"7466eeae-c194-42ea-ac6a-1fb3e8aa675a","Type":"ContainerDied","Data":"039972031a92faba3621d3699e668c2bc6059ee6aeddc3948cae13fe7188b2c4"} Oct 30 17:40:28.312650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.312623 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:28.419753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.419686 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\" (UniqueName: \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\") pod \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " Oct 30 17:40:28.419897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.419775 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\") pod \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " Oct 30 17:40:28.419946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.419908 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\") pod \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\" (UID: \"7466eeae-c194-42ea-ac6a-1fb3e8aa675a\") " Oct 30 17:40:28.420296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.420246 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca" pod "7466eeae-c194-42ea-ac6a-1fb3e8aa675a" (UID: "7466eeae-c194-42ea-ac6a-1fb3e8aa675a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:28.420681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.420659 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:28.421787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.421763 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca" pod "7466eeae-c194-42ea-ac6a-1fb3e8aa675a" (UID: "7466eeae-c194-42ea-ac6a-1fb3e8aa675a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:28.421866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.421797 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f" pod "7466eeae-c194-42ea-ac6a-1fb3e8aa675a" (UID: "7466eeae-c194-42ea-ac6a-1fb3e8aa675a"). InnerVolumeSpecName "kube-api-access-2ts2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:28.522056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.522027 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ts2f\" (UniqueName: \"kubernetes.io/projected/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-kube-api-access-2ts2f\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:28.522056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:28.522057 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7466eeae-c194-42ea-ac6a-1fb3e8aa675a-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:29.170061 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:29.170034 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" Oct 30 17:40:29.170289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:29.170029 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs" event={"ID":"7466eeae-c194-42ea-ac6a-1fb3e8aa675a","Type":"ContainerDied","Data":"0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384"} Oct 30 17:40:29.170289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:29.170156 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0894bb4b9ebee91fd3a7368a4475ceeb67d39a48831f36288a8ecccde9ace384" Oct 30 17:40:30.551308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.551243 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:40:30.587606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.587583 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:40:30.656232 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.656174 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.656621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.656515 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.656621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.656588 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.656831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.656637 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:40:30.657037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.656968 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.657196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657162 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.657439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657403 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.657676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657601 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.657676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657646 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.658109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657908 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.658109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.657951 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.658109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.658035 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.658437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.658402 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.658674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.658616 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.659051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.658903 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.659051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.658984 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.659051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.659015 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.659438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.659371 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\") pod \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\" (UID: \"bcb86078-c1be-4fd7-8dae-71231ed7b1ce\") " Oct 30 17:40:30.659674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.659622 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\") pod \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\" (UID: \"417f2c41-c04e-44fd-87c5-77fb7b1d6de7\") " Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.661790 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.661899 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.661984 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.662372 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.662429 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.662579 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.662788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.662773 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.663430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.663408 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.664070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.664038 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.664552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.664532 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.671574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671532 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.671574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671535 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:30.671574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671558 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.671735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671582 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "kube-api-access-wkj97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.671735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671590 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "kube-api-access-9smrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.671735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.671691 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token" pod "bcb86078-c1be-4fd7-8dae-71231ed7b1ce" (UID: "bcb86078-c1be-4fd7-8dae-71231ed7b1ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.672500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.672481 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.672726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.672707 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo" pod "417f2c41-c04e-44fd-87c5-77fb7b1d6de7" (UID: "417f2c41-c04e-44fd-87c5-77fb7b1d6de7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:30.761965 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.761884 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.761969 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.761996 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762021 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762041 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762112 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762133 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762169 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762179 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762357 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\") pod \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\" (UID: \"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5\") " Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762464 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.762550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762538 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.762879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762593 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.762879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.762597 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:30.763108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763080 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763104 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763118 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763128 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763137 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkj97\" (UniqueName: \"kubernetes.io/projected/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-kube-api-access-wkj97\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763146 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763155 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763162 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763172 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763179 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763189 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763197 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763205 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763212 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763219 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763225 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763232 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smrj\" (UniqueName: \"kubernetes.io/projected/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-kube-api-access-9smrj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763240 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763248 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763256 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.763248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763284 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.764309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763297 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/417f2c41-c04e-44fd-87c5-77fb7b1d6de7-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.764309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.763305 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb86078-c1be-4fd7-8dae-71231ed7b1ce-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.764565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.764541 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "kube-api-access-6w9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.764751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.764734 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:30.764751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.764740 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:30.764841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.764781 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy" pod "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" (UID: "b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:30.864744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.864711 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.864744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.864741 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.864962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.864752 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9kv\" (UniqueName: \"kubernetes.io/projected/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-kube-api-access-6w9kv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:30.864962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:30.864761 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:31.185753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.185718 4622 generic.go:356] "Generic (PLEG): container finished" podID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerID="73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d" exitCode=0 Oct 30 17:40:31.185959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.185776 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" Oct 30 17:40:31.185959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.185800 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" event={"ID":"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5","Type":"ContainerDied","Data":"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d"} Oct 30 17:40:31.185959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.185862 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm" event={"ID":"b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5","Type":"ContainerDied","Data":"33c6f4ace35af8df37f1e56f6f29ac3a5d7645fc6784b4bafe1a20fa2866a5fb"} Oct 30 17:40:31.185959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.185892 4622 scope.go:122] "RemoveContainer" containerID="73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d" Oct 30 17:40:31.187359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.187335 4622 generic.go:356] "Generic (PLEG): container finished" podID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerID="f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e" exitCode=0 Oct 30 17:40:31.187461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.187381 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" Oct 30 17:40:31.187461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.187420 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" event={"ID":"bcb86078-c1be-4fd7-8dae-71231ed7b1ce","Type":"ContainerDied","Data":"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e"} Oct 30 17:40:31.187588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.187466 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x" event={"ID":"bcb86078-c1be-4fd7-8dae-71231ed7b1ce","Type":"ContainerDied","Data":"eea79513b8cb6f56a25cd6ce55065e6b2dd5c88f878dace89b5ebdb158de86e1"} Oct 30 17:40:31.189032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.189009 4622 generic.go:356] "Generic (PLEG): container finished" podID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerID="f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255" exitCode=0 Oct 30 17:40:31.189147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.189063 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" Oct 30 17:40:31.189147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.189088 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" event={"ID":"417f2c41-c04e-44fd-87c5-77fb7b1d6de7","Type":"ContainerDied","Data":"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255"} Oct 30 17:40:31.189147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.189126 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8" event={"ID":"417f2c41-c04e-44fd-87c5-77fb7b1d6de7","Type":"ContainerDied","Data":"609adbde90a5ef3c78ca779b0427ee5640ce3aab8c17199aa46f2f59d13c73d5"} Oct 30 17:40:31.197786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.197769 4622 scope.go:122] "RemoveContainer" containerID="73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d" Oct 30 17:40:31.198082 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:31.198065 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d\": container with ID starting with 73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d not found: ID does not exist" containerID="73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d" Oct 30 17:40:31.198133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.198092 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d"} err="failed to get container status \"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d\": rpc error: code = NotFound desc = could not find container \"73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d\": container with ID starting with 73dc120c2f863cb8a878bbb215c4cb6019a2cc07487a6beb197926c3f50bea2d not found: ID does not exist" Oct 30 17:40:31.198133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.198108 4622 scope.go:122] "RemoveContainer" containerID="f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e" Oct 30 17:40:31.208254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.208238 4622 scope.go:122] "RemoveContainer" containerID="f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e" Oct 30 17:40:31.208511 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:31.208494 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e\": container with ID starting with f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e not found: ID does not exist" containerID="f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e" Oct 30 17:40:31.208578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.208524 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e"} err="failed to get container status \"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e\": rpc error: code = NotFound desc = could not find container \"f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e\": container with ID starting with f12611e28ac1d0c2138d269a2bcd8c6752865ec88a4cb095cfff968a87f83d6e not found: ID does not exist" Oct 30 17:40:31.208578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.208548 4622 scope.go:122] "RemoveContainer" containerID="f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255" Oct 30 17:40:31.217032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.217013 4622 scope.go:122] "RemoveContainer" containerID="f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255" Oct 30 17:40:31.217341 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:31.217312 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255\": container with ID starting with f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255 not found: ID does not exist" containerID="f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255" Oct 30 17:40:31.217430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.217351 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255"} err="failed to get container status \"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255\": rpc error: code = NotFound desc = could not find container \"f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255\": container with ID starting with f9edc55f3aee22ea2390ae71becf0c6de9aaa6e7cb4bd33f145b9f642eb34255 not found: ID does not exist" Oct 30 17:40:31.257096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.257069 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:40:31.269942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.269919 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-f7c4x"] Oct 30 17:40:31.294342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.294302 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:40:31.298508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.298484 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-csxwm"] Oct 30 17:40:31.333989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.333960 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:40:31.337553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:31.337533 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-56dp8"] Oct 30 17:40:32.376200 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:32.376169 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" path="/var/lib/kubelet/pods/417f2c41-c04e-44fd-87c5-77fb7b1d6de7/volumes" Oct 30 17:40:32.376679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:32.376664 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" path="/var/lib/kubelet/pods/b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5/volumes" Oct 30 17:40:32.377073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:32.377061 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" path="/var/lib/kubelet/pods/bcb86078-c1be-4fd7-8dae-71231ed7b1ce/volumes" Oct 30 17:40:33.192196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192162 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8860-25f504"] Oct 30 17:40:33.192786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192772 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" Oct 30 17:40:33.192836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192787 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" Oct 30 17:40:33.192836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192810 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7466eeae-c194-42ea-ac6a-1fb3e8aa675a" containerName="revoke-keys" Oct 30 17:40:33.192836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192815 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7466eeae-c194-42ea-ac6a-1fb3e8aa675a" containerName="revoke-keys" Oct 30 17:40:33.192836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192828 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" Oct 30 17:40:33.192836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192833 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192850 4622 cpu_manager.go:414] "containerMap: removing container" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192855 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192942 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb86078-c1be-4fd7-8dae-71231ed7b1ce" containerName="istio-proxy" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192954 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7466eeae-c194-42ea-ac6a-1fb3e8aa675a" containerName="revoke-keys" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192962 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a23bd3-25d9-43ad-8cc8-b0e3c6d7b3d5" containerName="istio-proxy" Oct 30 17:40:33.192989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.192968 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="417f2c41-c04e-44fd-87c5-77fb7b1d6de7" containerName="istio-proxy" Oct 30 17:40:33.197837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.197816 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:33.213420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.213396 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8860-25f504"] Oct 30 17:40:33.220541 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.220516 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5c6gd\"" Oct 30 17:40:33.295615 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.295580 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmmh\" (UniqueName: \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\") pod \"test-curl-8860-25f504\" (UID: \"1abded75-4ae4-445f-8564-6c41341dabfc\") " pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:33.396970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.396938 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmmh\" (UniqueName: \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\") pod \"test-curl-8860-25f504\" (UID: \"1abded75-4ae4-445f-8564-6c41341dabfc\") " pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:33.420044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.420017 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmmh\" (UniqueName: \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\") pod \"test-curl-8860-25f504\" (UID: \"1abded75-4ae4-445f-8564-6c41341dabfc\") " pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:33.513853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.513781 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:33.812929 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:33.812896 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1abded75_4ae4_445f_8564_6c41341dabfc.slice/crio-604591807b6f1bf231adf1ed011b5e10c29a23d3d0277468e5c328e4f6755ce4 WatchSource:0}: Error finding container 604591807b6f1bf231adf1ed011b5e10c29a23d3d0277468e5c328e4f6755ce4: Status 404 returned error can't find the container with id 604591807b6f1bf231adf1ed011b5e10c29a23d3d0277468e5c328e4f6755ce4 Oct 30 17:40:33.813075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:33.812981 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8860-25f504"] Oct 30 17:40:34.219407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:34.218232 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8860-25f504" event={"ID":"1abded75-4ae4-445f-8564-6c41341dabfc","Type":"ContainerStarted","Data":"32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71"} Oct 30 17:40:34.219407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:34.218311 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8860-25f504" event={"ID":"1abded75-4ae4-445f-8564-6c41341dabfc","Type":"ContainerStarted","Data":"604591807b6f1bf231adf1ed011b5e10c29a23d3d0277468e5c328e4f6755ce4"} Oct 30 17:40:34.285170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:34.285123 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8860-25f504" podStartSLOduration=1.285109057 podStartE2EDuration="1.285109057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:34.281522244 +0000 UTC m=+1662.058994199" watchObservedRunningTime="2026-10-30 17:40:34.285109057 +0000 UTC m=+1662.062581012" Oct 30 17:40:49.473348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.473312 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8860-25f504"] Oct 30 17:40:49.473928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.473516 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8860-25f504" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" containerName="test-curl-8860-25f504" containerID="cri-o://32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71" gracePeriod=2 Oct 30 17:40:49.474973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.474799 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8860-25f504"] Oct 30 17:40:49.487298 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:49.487249 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8860-25f504\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:49.716492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.716461 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w"] Oct 30 17:40:49.717061 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.717047 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" containerName="test-curl-8860-25f504" Oct 30 17:40:49.717116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.717062 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" containerName="test-curl-8860-25f504" Oct 30 17:40:49.717164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.717155 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" containerName="test-curl-8860-25f504" Oct 30 17:40:49.720977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.720957 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.742129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.742103 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w"] Oct 30 17:40:49.747599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.747576 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:40:49.747772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.747651 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:40:49.772536 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:49.772506 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8860-25f504\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:49.835689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.835659 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gwr\" (UniqueName: \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.835803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.835693 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.835850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.835819 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.937437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.937408 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gwr\" (UniqueName: \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.937567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.937441 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.937654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.937633 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.938290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.938239 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.939758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.939736 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:49.961189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:49.961165 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gwr\" (UniqueName: \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\") pod \"maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:50.034768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:50.034750 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:50.338064 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:50.338030 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03286223_b39f_4e0f_8e7a_d8ff67a7602e.slice/crio-0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5 WatchSource:0}: Error finding container 0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5: Status 404 returned error can't find the container with id 0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5 Oct 30 17:40:50.338517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:50.338494 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w"] Oct 30 17:40:51.347837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.347803 4622 generic.go:356] "Generic (PLEG): container finished" podID="03286223-b39f-4e0f-8e7a-d8ff67a7602e" containerID="a6bd2178a173370ee29b7b6d4c06226d222df77d9ad6760881bb3117a442ac4f" exitCode=0 Oct 30 17:40:51.348224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.347881 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" event={"ID":"03286223-b39f-4e0f-8e7a-d8ff67a7602e","Type":"ContainerDied","Data":"a6bd2178a173370ee29b7b6d4c06226d222df77d9ad6760881bb3117a442ac4f"} Oct 30 17:40:51.348224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.347962 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" event={"ID":"03286223-b39f-4e0f-8e7a-d8ff67a7602e","Type":"ContainerStarted","Data":"0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5"} Oct 30 17:40:51.676693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.676671 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:51.759611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.759552 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\" (UniqueName: \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\") pod \"1abded75-4ae4-445f-8564-6c41341dabfc\" (UID: \"1abded75-4ae4-445f-8564-6c41341dabfc\") " Oct 30 17:40:51.761532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.761509 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh" pod "1abded75-4ae4-445f-8564-6c41341dabfc" (UID: "1abded75-4ae4-445f-8564-6c41341dabfc"). InnerVolumeSpecName "kube-api-access-wdmmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:51.861376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:51.861344 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmmh\" (UniqueName: \"kubernetes.io/projected/1abded75-4ae4-445f-8564-6c41341dabfc-kube-api-access-wdmmh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.356035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.356001 4622 generic.go:356] "Generic (PLEG): container finished" podID="1abded75-4ae4-445f-8564-6c41341dabfc" containerID="32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71" exitCode=137 Oct 30 17:40:52.356492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.356044 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:52.356492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.356116 4622 scope.go:122] "RemoveContainer" containerID="32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71" Oct 30 17:40:52.366402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.366383 4622 scope.go:122] "RemoveContainer" containerID="32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71" Oct 30 17:40:52.366683 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:52.366661 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71\": container with ID starting with 32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71 not found: ID does not exist" containerID="32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71" Oct 30 17:40:52.366761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.366694 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71"} err="failed to get container status \"32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71\": rpc error: code = NotFound desc = could not find container \"32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71\": container with ID starting with 32e6ebb9545a5d77d6e098faf4f74013cd6600bc1770e4a01c22669b3b626c71 not found: ID does not exist" Oct 30 17:40:52.375389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.375369 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" path="/var/lib/kubelet/pods/1abded75-4ae4-445f-8564-6c41341dabfc/volumes" Oct 30 17:40:52.377964 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:52.377944 4622 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8860-25f504\" is forbidden: User \"system:node:ip-10-0-143-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-147.ec2.internal' and this object" podUID="1abded75-4ae4-445f-8564-6c41341dabfc" pod="opendatahub/test-curl-8860-25f504" Oct 30 17:40:52.511836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.511812 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:52.571491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.571463 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\") pod \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " Oct 30 17:40:52.571650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.571534 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\") pod \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " Oct 30 17:40:52.571650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.571578 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\" (UniqueName: \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\") pod \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\" (UID: \"03286223-b39f-4e0f-8e7a-d8ff67a7602e\") " Oct 30 17:40:52.571925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.571900 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca" pod "03286223-b39f-4e0f-8e7a-d8ff67a7602e" (UID: "03286223-b39f-4e0f-8e7a-d8ff67a7602e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:52.572080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.572067 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.573605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.573577 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca" pod "03286223-b39f-4e0f-8e7a-d8ff67a7602e" (UID: "03286223-b39f-4e0f-8e7a-d8ff67a7602e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:52.573704 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.573678 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr" pod "03286223-b39f-4e0f-8e7a-d8ff67a7602e" (UID: "03286223-b39f-4e0f-8e7a-d8ff67a7602e"). InnerVolumeSpecName "kube-api-access-x2gwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:52.674006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.673931 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gwr\" (UniqueName: \"kubernetes.io/projected/03286223-b39f-4e0f-8e7a-d8ff67a7602e-kube-api-access-x2gwr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:52.674006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:52.673966 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03286223-b39f-4e0f-8e7a-d8ff67a7602e-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:40:53.371147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:53.371121 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" Oct 30 17:40:53.371147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:53.371130 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w" event={"ID":"03286223-b39f-4e0f-8e7a-d8ff67a7602e","Type":"ContainerDied","Data":"0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5"} Oct 30 17:40:53.371612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:53.371168 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5bed48e4a7d97c82dc827c350cfe9db143f3acd0b3b7c6fb8d8cf0c975ceb5" Oct 30 17:40:54.220042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.220007 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:40:54.220699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.220680 4622 cpu_manager.go:414] "containerMap: removing container" podUID="03286223-b39f-4e0f-8e7a-d8ff67a7602e" containerName="revoke-keys" Oct 30 17:40:54.220699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.220700 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03286223-b39f-4e0f-8e7a-d8ff67a7602e" containerName="revoke-keys" Oct 30 17:40:54.220851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.220823 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="03286223-b39f-4e0f-8e7a-d8ff67a7602e" containerName="revoke-keys" Oct 30 17:40:54.225749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.225727 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.240658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.240578 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:40:54.245512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.245490 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.247635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.247613 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81e9336e-openshift-default-dockercfg-4mbp4\"" Oct 30 17:40:54.270250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.267253 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:40:54.275778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.275749 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:40:54.295380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295347 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.295587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295559 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.295679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295649 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.295775 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295755 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.295881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295852 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.295939 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.295922 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zmp\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.296056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.296035 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.296136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.296120 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.296224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.296208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397488 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397529 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397550 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397575 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397705 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397766 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397794 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397822 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397851 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397887 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.397956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397960 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrgd\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.397984 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398024 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zmp\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398044 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398061 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398096 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398194 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398231 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398247 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.398388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398302 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.399125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398445 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.399125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.398792 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.399125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.399003 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.400314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.400293 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.400429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.400414 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.421918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.421890 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.422223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.422201 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zmp\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.499215 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499188 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499224 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499339 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499379 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499414 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499452 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrgd\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499530 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499556 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499662 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499676 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499714 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.499868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.499761 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.500187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.500164 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.500324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.500308 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.501608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.501589 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.501712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.501693 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.522612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.522555 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrgd\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.523795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.523775 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-n27zg\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.533576 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:40:54.533551 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:40:54.541865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.541847 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:54.576174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.576148 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:54.859254 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:54.859222 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf099ae96_7fbc_4f8a_9514_323e9c2fbc04.slice/crio-538041c5ee541e8d12009142cc8fa47057b06d93e461c448191beab4ce7b9a84 WatchSource:0}: Error finding container 538041c5ee541e8d12009142cc8fa47057b06d93e461c448191beab4ce7b9a84: Status 404 returned error can't find the container with id 538041c5ee541e8d12009142cc8fa47057b06d93e461c448191beab4ce7b9a84 Oct 30 17:40:54.860298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.860279 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:40:54.862097 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.862067 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:54.862175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.862137 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:54.862175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.862168 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:54.873525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.873498 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:40:54.878886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.878865 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:54.898151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.898117 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:40:54.900026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.899212 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-49e155-serving-cert\"" Oct 30 17:40:54.909668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.904911 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:40:54.910782 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:54.910755 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1028a70e_d228_4f40_b7a3_72b23f9c1c1d.slice/crio-acf19b3b8849c42789aa9ec4d1295701566fe7173aee0a48b445362376e2f715 WatchSource:0}: Error finding container acf19b3b8849c42789aa9ec4d1295701566fe7173aee0a48b445362376e2f715: Status 404 returned error can't find the container with id acf19b3b8849c42789aa9ec4d1295701566fe7173aee0a48b445362376e2f715 Oct 30 17:40:54.917543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.914470 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:54.917543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.914550 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:54.917543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:54.914599 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:40:55.005856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.005829 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.006076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.006057 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrj84\" (UniqueName: \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.006168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.006124 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.006230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.006190 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.108754 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.108327 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrj84\" (UniqueName: \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.109413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.109388 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.113060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.113003 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.113206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.113187 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.113458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.113440 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.117034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.116991 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.123305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.118459 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.139284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.138384 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrj84\" (UniqueName: \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\") pod \"maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.215610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.215577 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:55.389970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.389105 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" event={"ID":"1028a70e-d228-4f40-b7a3-72b23f9c1c1d","Type":"ContainerStarted","Data":"822698d9dcba42106e15e178cc563ee781febfb5a3f052f9fb7e45cc8e019eb6"} Oct 30 17:40:55.389970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.389175 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" event={"ID":"1028a70e-d228-4f40-b7a3-72b23f9c1c1d","Type":"ContainerStarted","Data":"acf19b3b8849c42789aa9ec4d1295701566fe7173aee0a48b445362376e2f715"} Oct 30 17:40:55.391254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.391226 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" event={"ID":"f099ae96-7fbc-4f8a-9514-323e9c2fbc04","Type":"ContainerStarted","Data":"2f56e8fb4b61aa63346cfdef9b9a2dfe401af1cc077315396acb122c3b5f2613"} Oct 30 17:40:55.391403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.391308 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" event={"ID":"f099ae96-7fbc-4f8a-9514-323e9c2fbc04","Type":"ContainerStarted","Data":"538041c5ee541e8d12009142cc8fa47057b06d93e461c448191beab4ce7b9a84"} Oct 30 17:40:55.436997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.436937 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podStartSLOduration=1.4369180080000001 podStartE2EDuration="1.436918008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:55.435213935 +0000 UTC m=+1683.212685915" watchObservedRunningTime="2026-10-30 17:40:55.436918008 +0000 UTC m=+1683.214389965" Oct 30 17:40:55.546441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.546400 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:40:55.552288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.550384 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:40:55.552288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.550446 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:40:55.577181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.576804 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:40:55.598373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.595989 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podStartSLOduration=1.595967821 podStartE2EDuration="1.595967821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:55.502568935 +0000 UTC m=+1683.280040890" watchObservedRunningTime="2026-10-30 17:40:55.595967821 +0000 UTC m=+1683.373439777" Oct 30 17:40:55.600561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.600533 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:40:55.606296 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:55.604378 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118c0be6_649b_4f1f_bc67_4deca469ebb1.slice/crio-f8293a608233f787e55ece02626e8c5fcb3e0ce5afb4f4dc1ee4e1a767677251 WatchSource:0}: Error finding container f8293a608233f787e55ece02626e8c5fcb3e0ce5afb4f4dc1ee4e1a767677251: Status 404 returned error can't find the container with id f8293a608233f787e55ece02626e8c5fcb3e0ce5afb4f4dc1ee4e1a767677251 Oct 30 17:40:55.606296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:55.606080 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:40:56.435012 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.434733 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" event={"ID":"118c0be6-649b-4f1f-bc67-4deca469ebb1","Type":"ContainerStarted","Data":"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c"} Oct 30 17:40:56.435012 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.434782 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" event={"ID":"118c0be6-649b-4f1f-bc67-4deca469ebb1","Type":"ContainerStarted","Data":"f8293a608233f787e55ece02626e8c5fcb3e0ce5afb4f4dc1ee4e1a767677251"} Oct 30 17:40:56.435216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.435160 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:40:56.547462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.547430 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:40:56.548093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.548045 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:40:56.615300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.613346 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:40:56.615300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:56.613411 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:40:57.195286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.194999 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" podStartSLOduration=2.797971233 podStartE2EDuration="3.194976646s" totalImagesPullingTime="397.005413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:56.494529206 +0000 UTC m=+1684.272001161" watchObservedRunningTime="2026-10-30 17:40:57.194976646 +0000 UTC m=+1684.972448602" Oct 30 17:40:57.198300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.197791 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:40:57.206391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.205901 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:40:57.206391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.206002 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.224692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.221513 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cc4e96-serving-cert\"" Oct 30 17:40:57.300515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.300473 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.300728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.300588 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8s9\" (UniqueName: \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.300802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.300738 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.300802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.300784 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.401978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.401944 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.402161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.402096 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8s9\" (UniqueName: \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.402319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.402300 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.402388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.402359 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.403397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.403376 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.405108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.405081 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.405559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.405540 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.426468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.425709 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8s9\" (UniqueName: \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\") pod \"maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.523605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.523504 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:57.552749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.552715 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:40:57.553343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.553310 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:40:57.595287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.584236 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:40:57.595287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.584306 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:40:57.939133 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:40:57.939104 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9af145_6fd5_414b_88a8_eaeff045fa5c.slice/crio-ec002a8ac10ddbb373ff454fd80ddaff0441641d9b2bc311bf5923e27352b92a WatchSource:0}: Error finding container ec002a8ac10ddbb373ff454fd80ddaff0441641d9b2bc311bf5923e27352b92a: Status 404 returned error can't find the container with id ec002a8ac10ddbb373ff454fd80ddaff0441641d9b2bc311bf5923e27352b92a Oct 30 17:40:57.940120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:57.940096 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:40:58.555280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:58.554544 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:40:58.555280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:58.554609 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:40:58.592288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:58.592054 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:40:58.592288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:58.592115 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:40:58.671711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:58.671678 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" event={"ID":"fa9af145-6fd5-414b-88a8-eaeff045fa5c","Type":"ContainerStarted","Data":"ec002a8ac10ddbb373ff454fd80ddaff0441641d9b2bc311bf5923e27352b92a"} Oct 30 17:40:59.560287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.548732 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:40:59.560287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.548823 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:40:59.583288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.577619 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:40:59.583288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.577678 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:40:59.699614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.699563 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" event={"ID":"fa9af145-6fd5-414b-88a8-eaeff045fa5c","Type":"ContainerStarted","Data":"9d503a211b6b11141e65dbf027b7a4bb80281bc2c5aa0188f97cbb8e77a6d577"} Oct 30 17:40:59.699766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.699753 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:40:59.776288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:40:59.775433 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" podStartSLOduration=2.306952841 podStartE2EDuration="2.775413985s" totalImagesPullingTime="468.461144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:59.753960361 +0000 UTC m=+1687.531432318" watchObservedRunningTime="2026-10-30 17:40:59.775413985 +0000 UTC m=+1687.552885941" Oct 30 17:41:00.546542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:00.544584 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:41:00.546542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:00.544642 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:41:00.578284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:00.577624 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:41:00.578284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:00.577684 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:41:01.542670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:01.542629 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 17:41:01.542831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:01.542699 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 17:41:01.577587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:01.577547 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 17:41:01.577747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:01.577615 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 17:41:02.507613 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.507583 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:41:02.547744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.547714 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:41:02.548103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.548076 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:41:02.548881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.548846 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:41:02.588174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.588147 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:41:02.589562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.589540 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:41:02.596077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:02.596056 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:41:03.129457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:03.129424 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w"] Oct 30 17:41:03.133664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:03.133637 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81e9336e-z8c8w"] Oct 30 17:41:04.377228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:04.377185 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03286223-b39f-4e0f-8e7a-d8ff67a7602e" path="/var/lib/kubelet/pods/03286223-b39f-4e0f-8e7a-d8ff67a7602e/volumes" Oct 30 17:41:05.713697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:05.713669 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:41:09.301022 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.300984 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:41:09.306307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.306276 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.324658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.324627 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.324891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.324858 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.324938 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.324981 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.325012 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.325075 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.325095 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.325166 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.325326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.325203 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bfxm\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.330057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.330027 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:41:09.426126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426092 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426139 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426186 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426214 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426325 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426420 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426454 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426581 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426583 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426661 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bfxm\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426671 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.426825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.426810 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.427029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.427006 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.427084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.427038 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.428671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.428650 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.428766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.428736 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.449919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.449892 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.450077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.450059 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bfxm\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\") pod \"e2e-mt-81e9336e-openshift-default-5db946866f-d8929\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.633635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.633547 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:09.950677 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:41:09.950642 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a68059b_997b_4fd9_baa7_58dc1493a28d.slice/crio-c05f4ce9a3770ba7061167ecb26e232a27d8eb23b88a82c5b7e03799d2ea5451 WatchSource:0}: Error finding container c05f4ce9a3770ba7061167ecb26e232a27d8eb23b88a82c5b7e03799d2ea5451: Status 404 returned error can't find the container with id c05f4ce9a3770ba7061167ecb26e232a27d8eb23b88a82c5b7e03799d2ea5451 Oct 30 17:41:09.951089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.951067 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:41:09.953395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.953360 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:09.953515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.953441 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:09.953515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:09.953489 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:10.801203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:10.801165 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" event={"ID":"1a68059b-997b-4fd9-baa7-58dc1493a28d","Type":"ContainerStarted","Data":"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a"} Oct 30 17:41:10.801610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:10.801221 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" event={"ID":"1a68059b-997b-4fd9-baa7-58dc1493a28d","Type":"ContainerStarted","Data":"c05f4ce9a3770ba7061167ecb26e232a27d8eb23b88a82c5b7e03799d2ea5451"} Oct 30 17:41:10.875945 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:10.875890 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podStartSLOduration=1.875875636 podStartE2EDuration="1.875875636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:10.859911629 +0000 UTC m=+1698.637383586" watchObservedRunningTime="2026-10-30 17:41:10.875875636 +0000 UTC m=+1698.653347590" Oct 30 17:41:11.634639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:11.634603 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:11.636392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:11.636363 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:11.636569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:11.636418 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:12.634348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:12.634303 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:12.634790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:12.634362 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:13.634362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:13.634325 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:13.634813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:13.634378 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:14.634720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:14.634686 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:14.635151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:14.634741 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:15.634817 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:15.634782 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:15.635213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:15.634863 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:16.634388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:16.634350 4622 patch_prober.go:29] interesting pod/e2e-mt-81e9336e-openshift-default-5db946866f-d8929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:41:16.634570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:16.634427 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 17:41:17.639607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:17.639577 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:17.640079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:17.640063 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:17.640736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:17.640711 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:22.728780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:22.728726 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:23.037285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:23.036360 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf"] Oct 30 17:41:23.054334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:23.053222 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3277aa70-nfzbf"] Oct 30 17:41:23.199296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:23.196350 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:41:23.199296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:23.196430 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:41:24.379474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:24.379441 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2f86bf-fb41-4f6a-9a2c-76d10ad676db" path="/var/lib/kubelet/pods/4e2f86bf-fb41-4f6a-9a2c-76d10ad676db/volumes" Oct 30 17:41:24.535078 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:41:24.535038 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:41:26.096323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:26.094732 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:41:26.096323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:26.094994 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" containerID="cri-o://270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7" gracePeriod=30 Oct 30 17:41:31.414729 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.414694 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:41:31.476713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.476678 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.476895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.476780 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.476977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.476918 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.476977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.476929 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:31.476977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.476962 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477052 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477087 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477198 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477182 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477245 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477392 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\") pod \"adac9297-76e3-48da-aa15-3e66db6dec3f\" (UID: \"adac9297-76e3-48da-aa15-3e66db6dec3f\") " Oct 30 17:41:31.477940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.477912 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:31.478104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478086 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:31.478428 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478406 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478673 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478703 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adac9297-76e3-48da-aa15-3e66db6dec3f-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478719 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.478732 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.479322 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:31.479435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.479412 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:31.480988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.480963 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:31.481150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.481131 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "kube-api-access-d9bwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:31.482415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.482369 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo" pod "adac9297-76e3-48da-aa15-3e66db6dec3f" (UID: "adac9297-76e3-48da-aa15-3e66db6dec3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:31.581304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.580687 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.581304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.580725 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.581304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.580739 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adac9297-76e3-48da-aa15-3e66db6dec3f-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.581304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.580752 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adac9297-76e3-48da-aa15-3e66db6dec3f-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:31.581304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:31.580766 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9bwh\" (UniqueName: \"kubernetes.io/projected/adac9297-76e3-48da-aa15-3e66db6dec3f-kube-api-access-d9bwh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:32.072228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.072193 4622 generic.go:356] "Generic (PLEG): container finished" podID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerID="2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4" exitCode=0 Oct 30 17:41:32.072408 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.072296 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerDied","Data":"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4"} Oct 30 17:41:32.072787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.072767 4622 scope.go:122] "RemoveContainer" containerID="2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4" Oct 30 17:41:32.074151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074122 4622 generic.go:356] "Generic (PLEG): container finished" podID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerID="270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7" exitCode=0 Oct 30 17:41:32.074521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074185 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" Oct 30 17:41:32.074521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074207 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerDied","Data":"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7"} Oct 30 17:41:32.074521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074298 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g" event={"ID":"adac9297-76e3-48da-aa15-3e66db6dec3f","Type":"ContainerDied","Data":"3c16360ca2b67457e2be1ddaf1556d2777b171bfd0427414d7d4e27acbe57921"} Oct 30 17:41:32.074521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074334 4622 scope.go:122] "RemoveContainer" containerID="270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7" Oct 30 17:41:32.074777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074558 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:32.074777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074632 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:32.074777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.074677 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:32.089729 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.089707 4622 scope.go:122] "RemoveContainer" containerID="88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc" Oct 30 17:41:32.098905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.098884 4622 scope.go:122] "RemoveContainer" containerID="270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7" Oct 30 17:41:32.098905 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:41:32.099144 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7\": container with ID starting with 270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7 not found: ID does not exist" containerID="270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7" Oct 30 17:41:32.098905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.099178 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7"} err="failed to get container status \"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7\": rpc error: code = NotFound desc = could not find container \"270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7\": container with ID starting with 270b7e6c1923377d9cb93829cc7dfda94be53d4b0fcebea243971f60d20424a7 not found: ID does not exist" Oct 30 17:41:32.098905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.099200 4622 scope.go:122] "RemoveContainer" containerID="88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc" Oct 30 17:41:32.099513 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:41:32.099473 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc\": container with ID starting with 88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc not found: ID does not exist" containerID="88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc" Oct 30 17:41:32.099562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.099521 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc"} err="failed to get container status \"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc\": rpc error: code = NotFound desc = could not find container \"88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc\": container with ID starting with 88ff247d26e901ab7d6c35513bd762884b0c8940f515d9a13f0eaa9f3af7e5dc not found: ID does not exist" Oct 30 17:41:32.159103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.159079 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:41:32.174722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.171194 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d1c973-openshift-default-7b4d897445-q5g4g"] Oct 30 17:41:32.375185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:32.375150 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" path="/var/lib/kubelet/pods/adac9297-76e3-48da-aa15-3e66db6dec3f/volumes" Oct 30 17:41:33.083161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:33.083125 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerStarted","Data":"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a"} Oct 30 17:41:33.965292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:33.960556 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:41:33.966585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:33.966560 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 17:41:33.966714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:33.966606 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 17:41:34.090477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:34.090443 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:41:34.965321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:34.954667 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 17:41:34.965321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:34.954727 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 17:41:35.960282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:35.957712 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 17:41:35.960282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:35.957778 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 17:41:36.956285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:36.954620 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 17:41:36.956285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:36.954690 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 17:41:37.957619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:37.957583 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 17:41:37.958211 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:37.958178 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 17:41:38.959828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:38.959798 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:41:38.960826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:38.960804 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:41:41.655900 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.655865 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:41:41.656952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.656928 4622 cpu_manager.go:414] "containerMap: removing container" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.656952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.656954 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.657104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.656977 4622 cpu_manager.go:414] "containerMap: removing container" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.657104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.656985 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.657183 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.657171 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.657216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.657188 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="adac9297-76e3-48da-aa15-3e66db6dec3f" containerName="istio-proxy" Oct 30 17:41:41.662981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.662960 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.682579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.682487 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-38f445-openshift-default-dockercfg-x7t6p\"" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683278 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683350 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683386 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683462 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683568 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683625 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683668 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683715 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.685283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.683775 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qmh\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.701521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.701492 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:41:41.785343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.785298 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.785699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.785679 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.785863 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.785846 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.785991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.785976 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.786127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.786111 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.786280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.786249 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qmh\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.786576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.786556 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.786726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.786710 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.786883 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.786865 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.793680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.787904 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.793884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.790542 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.793992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.793553 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.794101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.793791 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.808574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.807568 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.809066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.809043 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.817344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.817320 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qmh\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.819090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.819024 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.856049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.856015 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\") pod \"e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:41.986625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:41.986541 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:42.366449 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:41:42.366416 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e380bc6_eabc_4c06_8a34_ce1e5919c599.slice/crio-9ad575c4dbde9846590f293151d9d3fb61beb1aac42fef89d1d843670135ce81 WatchSource:0}: Error finding container 9ad575c4dbde9846590f293151d9d3fb61beb1aac42fef89d1d843670135ce81: Status 404 returned error can't find the container with id 9ad575c4dbde9846590f293151d9d3fb61beb1aac42fef89d1d843670135ce81 Oct 30 17:41:42.370324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:42.370256 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:42.370451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:42.370369 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:42.370451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:42.370408 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:41:42.383406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:42.383372 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:41:43.170856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:43.170821 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerStarted","Data":"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808"} Oct 30 17:41:43.171248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:43.170887 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerStarted","Data":"9ad575c4dbde9846590f293151d9d3fb61beb1aac42fef89d1d843670135ce81"} Oct 30 17:41:43.233988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:43.233927 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podStartSLOduration=2.233902664 podStartE2EDuration="2.233902664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:43.227395665 +0000 UTC m=+1731.004867621" watchObservedRunningTime="2026-10-30 17:41:43.233902664 +0000 UTC m=+1731.011374621" Oct 30 17:41:43.987738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:43.987706 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:43.993563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:43.993537 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:44.178611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:44.178578 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:44.179650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:44.179627 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:41:53.964834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.964758 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:41:53.965303 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.964969 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" containerID="cri-o://822698d9dcba42106e15e178cc563ee781febfb5a3f052f9fb7e45cc8e019eb6" gracePeriod=30 Oct 30 17:41:53.978372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.978318 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:41:53.978614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.978569 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" containerID="cri-o://2f56e8fb4b61aa63346cfdef9b9a2dfe401af1cc077315396acb122c3b5f2613" gracePeriod=30 Oct 30 17:41:53.992979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.992949 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:41:53.993241 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:53.993214 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" containerID="cri-o://ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a" gracePeriod=30 Oct 30 17:41:54.538575 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:41:54.538538 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:41:59.300678 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.300645 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:59.315563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.315535 4622 generic.go:356] "Generic (PLEG): container finished" podID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerID="ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a" exitCode=0 Oct 30 17:41:59.315752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.315598 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" Oct 30 17:41:59.315861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.315795 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" event={"ID":"1a68059b-997b-4fd9-baa7-58dc1493a28d","Type":"ContainerDied","Data":"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a"} Oct 30 17:41:59.315861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.315844 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929" event={"ID":"1a68059b-997b-4fd9-baa7-58dc1493a28d","Type":"ContainerDied","Data":"c05f4ce9a3770ba7061167ecb26e232a27d8eb23b88a82c5b7e03799d2ea5451"} Oct 30 17:41:59.316034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.315875 4622 scope.go:122] "RemoveContainer" containerID="ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a" Oct 30 17:41:59.317323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.317255 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:41:59.317494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.317470 4622 generic.go:356] "Generic (PLEG): container finished" podID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerID="822698d9dcba42106e15e178cc563ee781febfb5a3f052f9fb7e45cc8e019eb6" exitCode=0 Oct 30 17:41:59.317609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.317562 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" event={"ID":"1028a70e-d228-4f40-b7a3-72b23f9c1c1d","Type":"ContainerDied","Data":"822698d9dcba42106e15e178cc563ee781febfb5a3f052f9fb7e45cc8e019eb6"} Oct 30 17:41:59.319512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.319488 4622 generic.go:356] "Generic (PLEG): container finished" podID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerID="2f56e8fb4b61aa63346cfdef9b9a2dfe401af1cc077315396acb122c3b5f2613" exitCode=0 Oct 30 17:41:59.319608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.319545 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" event={"ID":"f099ae96-7fbc-4f8a-9514-323e9c2fbc04","Type":"ContainerDied","Data":"2f56e8fb4b61aa63346cfdef9b9a2dfe401af1cc077315396acb122c3b5f2613"} Oct 30 17:41:59.330039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.330021 4622 scope.go:122] "RemoveContainer" containerID="ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a" Oct 30 17:41:59.330536 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:41:59.330512 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a\": container with ID starting with ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a not found: ID does not exist" containerID="ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a" Oct 30 17:41:59.330637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.330544 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a"} err="failed to get container status \"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a\": rpc error: code = NotFound desc = could not find container \"ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a\": container with ID starting with ed5f7761f81e9cbb9099bd8ad45ab86188a799e632cb46c413c825093f9aa80a not found: ID does not exist" Oct 30 17:41:59.330637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.330564 4622 scope.go:122] "RemoveContainer" containerID="822698d9dcba42106e15e178cc563ee781febfb5a3f052f9fb7e45cc8e019eb6" Oct 30 17:41:59.356584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.356566 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:41:59.400991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.400961 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.401137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401024 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.401137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401089 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401119 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401174 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.401331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401221 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401283 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.401331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401295 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.401331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401310 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401519 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.401649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401589 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.401703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401670 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401685 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.401813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401748 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.401871 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401843 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.401932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401879 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.401932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401919 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402038 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401956 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402038 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.401990 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402044 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402071 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402183 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402225 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402353 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.402462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402400 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.402462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402413 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402427 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.402462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402443 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.402677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402484 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402552 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402611 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402623 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.402677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402638 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.402935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402673 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\") pod \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\" (UID: \"1028a70e-d228-4f40-b7a3-72b23f9c1c1d\") " Oct 30 17:41:59.402935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402704 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\") pod \"1a68059b-997b-4fd9-baa7-58dc1493a28d\" (UID: \"1a68059b-997b-4fd9-baa7-58dc1493a28d\") " Oct 30 17:41:59.402935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402737 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\") pod \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\" (UID: \"f099ae96-7fbc-4f8a-9514-323e9c2fbc04\") " Oct 30 17:41:59.402935 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.402844 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.403176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403118 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.403664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403641 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403668 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403683 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403696 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403708 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403720 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403734 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403746 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403757 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.403769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.403769 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a68059b-997b-4fd9-baa7-58dc1493a28d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.404245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.404138 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.405414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.404824 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.405414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.405161 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:59.406365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.406010 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.406365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.406010 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "kube-api-access-vmrgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.406365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.406197 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.407179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.406808 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.407179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.406878 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.407179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.407066 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.407737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.407498 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:59.407974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.407914 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token" pod "1028a70e-d228-4f40-b7a3-72b23f9c1c1d" (UID: "1028a70e-d228-4f40-b7a3-72b23f9c1c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.408815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.408723 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.410692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.410239 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.411074 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.411041 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm" pod "1a68059b-997b-4fd9-baa7-58dc1493a28d" (UID: "1a68059b-997b-4fd9-baa7-58dc1493a28d"). InnerVolumeSpecName "kube-api-access-6bfxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.411167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.411134 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:59.417448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.417426 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp" pod "f099ae96-7fbc-4f8a-9514-323e9c2fbc04" (UID: "f099ae96-7fbc-4f8a-9514-323e9c2fbc04"). InnerVolumeSpecName "kube-api-access-b5zmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504819 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504846 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504856 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrgd\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-kube-api-access-vmrgd\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504866 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504875 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504882 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504890 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.504890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504897 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zmp\" (UniqueName: \"kubernetes.io/projected/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-kube-api-access-b5zmp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504907 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504919 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504930 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504939 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bfxm\" (UniqueName: \"kubernetes.io/projected/1a68059b-997b-4fd9-baa7-58dc1493a28d-kube-api-access-6bfxm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504946 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504954 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a68059b-997b-4fd9-baa7-58dc1493a28d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504961 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504969 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1028a70e-d228-4f40-b7a3-72b23f9c1c1d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.505282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.504977 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f099ae96-7fbc-4f8a-9514-323e9c2fbc04-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:41:59.665196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.665159 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:41:59.669170 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:41:59.669143 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-d8929"] Oct 30 17:42:00.328491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.328458 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" Oct 30 17:42:00.328491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.328462 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg" event={"ID":"1028a70e-d228-4f40-b7a3-72b23f9c1c1d","Type":"ContainerDied","Data":"acf19b3b8849c42789aa9ec4d1295701566fe7173aee0a48b445362376e2f715"} Oct 30 17:42:00.330019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.329995 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" Oct 30 17:42:00.330164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.330024 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk" event={"ID":"f099ae96-7fbc-4f8a-9514-323e9c2fbc04","Type":"ContainerDied","Data":"538041c5ee541e8d12009142cc8fa47057b06d93e461c448191beab4ce7b9a84"} Oct 30 17:42:00.330164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.330080 4622 scope.go:122] "RemoveContainer" containerID="2f56e8fb4b61aa63346cfdef9b9a2dfe401af1cc077315396acb122c3b5f2613" Oct 30 17:42:00.375414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.375388 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" path="/var/lib/kubelet/pods/1a68059b-997b-4fd9-baa7-58dc1493a28d/volumes" Oct 30 17:42:00.382387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.382362 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:42:00.385977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.385958 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-n27zg"] Oct 30 17:42:00.426050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.426023 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:42:00.433248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:00.433223 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81e9336e-openshift-default-5db946866f-zqlxk"] Oct 30 17:42:02.375502 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:02.375472 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" path="/var/lib/kubelet/pods/1028a70e-d228-4f40-b7a3-72b23f9c1c1d/volumes" Oct 30 17:42:02.375948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:02.375934 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" path="/var/lib/kubelet/pods/f099ae96-7fbc-4f8a-9514-323e9c2fbc04/volumes" Oct 30 17:42:07.297699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.297663 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:42:07.299212 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299182 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" Oct 30 17:42:07.299212 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299213 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" Oct 30 17:42:07.299489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299322 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" Oct 30 17:42:07.299489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299333 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" Oct 30 17:42:07.299489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299352 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" Oct 30 17:42:07.299489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299360 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" Oct 30 17:42:07.299697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299514 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1028a70e-d228-4f40-b7a3-72b23f9c1c1d" containerName="istio-proxy" Oct 30 17:42:07.299697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299526 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f099ae96-7fbc-4f8a-9514-323e9c2fbc04" containerName="istio-proxy" Oct 30 17:42:07.299697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.299544 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a68059b-997b-4fd9-baa7-58dc1493a28d" containerName="istio-proxy" Oct 30 17:42:07.304880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.304855 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.324017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.323985 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:42:07.326498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.326472 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f8b48d-openshift-default-dockercfg-4jlcw\"" Oct 30 17:42:07.434001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.433898 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g989t\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434045 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434157 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434252 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434352 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434412 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434555 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434588 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.434641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.434615 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537316 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g989t\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537410 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537474 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537531 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537571 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537623 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537700 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537731 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537754 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537874 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.537933 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.537924 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.538141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.538122 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.538342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.538239 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.538576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.538555 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.540850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.540824 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.540969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.540928 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.563132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.563082 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.563768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.563744 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g989t\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\") pod \"e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.628917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.628880 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:07.955794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.955725 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:42:07.966496 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:42:07.966459 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7604ad68_9342_45d8_be4d_4af284543004.slice/crio-d9e3aa1cfc1d22650fff714de2374a1308ae192517cddf7da995332d0d6083c7 WatchSource:0}: Error finding container d9e3aa1cfc1d22650fff714de2374a1308ae192517cddf7da995332d0d6083c7: Status 404 returned error can't find the container with id d9e3aa1cfc1d22650fff714de2374a1308ae192517cddf7da995332d0d6083c7 Oct 30 17:42:07.969292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.969237 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:07.969377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.969340 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:07.969377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:07.969373 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:08.403686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:08.403646 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerStarted","Data":"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da"} Oct 30 17:42:08.404087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:08.403712 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerStarted","Data":"d9e3aa1cfc1d22650fff714de2374a1308ae192517cddf7da995332d0d6083c7"} Oct 30 17:42:08.462205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:08.462151 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podStartSLOduration=1.462135771 podStartE2EDuration="1.462135771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:08.459244546 +0000 UTC m=+1756.236716500" watchObservedRunningTime="2026-10-30 17:42:08.462135771 +0000 UTC m=+1756.239607726" Oct 30 17:42:08.629454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:08.629417 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:09.635966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:09.635938 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:10.426557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:10.426526 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:10.429453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:10.429423 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:42:24.534122 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:42:24.534086 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:42:51.793645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.793588 4622 generic.go:356] "Generic (PLEG): container finished" podID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerID="51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808" exitCode=0 Oct 30 17:42:51.794523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.794488 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerDied","Data":"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808"} Oct 30 17:42:51.795226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.795203 4622 scope.go:122] "RemoveContainer" containerID="51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808" Oct 30 17:42:51.815550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.815504 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:51.815761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.815743 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:51.815885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:51.815871 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:52.411069 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.411038 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:42:52.414555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.414521 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.469453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.469421 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.469835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.469805 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.469963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.469895 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.469963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.469929 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:42:52.470088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.469937 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.470172 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.470154 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.470403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.470376 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsqx\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.470511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.470461 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.473321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.470588 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.473321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.470634 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.515658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.515624 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-74762d-openshift-default-dockercfg-92brx\"" Oct 30 17:42:52.571519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571487 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571532 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571553 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571623 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571724 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsqx\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571791 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571862 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.571966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571914 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571974 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.571913 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.572044 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.572296 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572398 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.572377 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.572445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.572420 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.574234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.574213 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.574376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.574358 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.599508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.599473 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsqx\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.605555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.605515 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\") pod \"e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.731740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.731639 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:52.804929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.804892 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerStarted","Data":"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1"} Oct 30 17:42:52.987393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.987305 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:42:52.987555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.987530 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:52.987620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:52.987598 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:53.099769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.099455 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:42:53.100027 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:42:53.099986 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb111a1d6_5c4a_4c3a_b6ff_62313e762b6d.slice/crio-0101fcd8409565790c669a3a552c16426dafe2b34136ea496c7ccc73fa578850 WatchSource:0}: Error finding container 0101fcd8409565790c669a3a552c16426dafe2b34136ea496c7ccc73fa578850: Status 404 returned error can't find the container with id 0101fcd8409565790c669a3a552c16426dafe2b34136ea496c7ccc73fa578850 Oct 30 17:42:53.103946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.103907 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:53.104052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.103999 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:53.104131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.104058 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:42:53.814084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.814048 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" event={"ID":"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d","Type":"ContainerStarted","Data":"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb"} Oct 30 17:42:53.814516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.814110 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" event={"ID":"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d","Type":"ContainerStarted","Data":"0101fcd8409565790c669a3a552c16426dafe2b34136ea496c7ccc73fa578850"} Oct 30 17:42:53.868947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.868762 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podStartSLOduration=1.868739271 podStartE2EDuration="1.868739271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:53.864557946 +0000 UTC m=+1801.642029903" watchObservedRunningTime="2026-10-30 17:42:53.868739271 +0000 UTC m=+1801.646211226" Oct 30 17:42:53.987851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.987818 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:53.988055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:53.987874 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:54.067847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:54.067746 4622 scope.go:122] "RemoveContainer" containerID="78295f22f3075ce7a3fcc54e15e7fb1d697bb61302a8d27bad7c84509acf8eb8" Oct 30 17:42:54.769287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:54.758199 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:42:54.817284 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:42:54.795469 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:42:54.817284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:54.795633 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:54.817284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:54.799374 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:42:55.021162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.021055 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:55.021162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.021120 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:55.732706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.732659 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:55.732875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.732733 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:42:55.996346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.993713 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:55.996346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:55.993776 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:56.739986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:56.738357 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:56.739986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:56.738407 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:42:56.994369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:56.992843 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:56.994369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:56.992902 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:57.732474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:57.732429 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:57.732976 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:57.732489 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:42:57.990383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:57.989752 4622 patch_prober.go:29] interesting pod/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 17:42:57.990383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:57.989829 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 17:42:58.732105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:58.732062 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:58.732269 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:58.732133 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:42:58.992725 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:58.992642 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:42:58.993189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:58.993032 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:42:58.993774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:58.993754 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:42:59.732087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:59.732047 4622 patch_prober.go:29] interesting pod/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:42:59.732288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:42:59.732115 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 17:43:00.737240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:00.737199 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:43:00.737743 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:00.737693 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:43:00.738519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:00.738497 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:43:24.535824 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:43:24.535733 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:43:35.437184 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.437098 4622 generic.go:356] "Generic (PLEG): container finished" podID="7604ad68-9342-45d8-be4d-4af284543004" containerID="dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da" exitCode=0 Oct 30 17:43:35.437742 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.437171 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerDied","Data":"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da"} Oct 30 17:43:35.437815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.437773 4622 scope.go:122] "RemoveContainer" containerID="dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da" Oct 30 17:43:35.439577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.439551 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:35.439679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.439620 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:35.439679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:35.439656 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:36.472798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.472759 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerStarted","Data":"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43"} Oct 30 17:43:36.629854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.629819 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:43:36.631211 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.631184 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:36.631400 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.631242 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:36.684027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.683998 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:43:36.687752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.687731 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.716624 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.716579 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:43:36.796909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.796884 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-874b83-openshift-default-dockercfg-nmtqj\"" Oct 30 17:43:36.870822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.870778 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.870984 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgdn6\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871027 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871090 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871125 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871149 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871329 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871448 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.871544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.871525 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972419 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972488 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972587 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgdn6\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972611 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972652 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972635 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972676 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972708 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972768 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.972856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972848 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.973065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972944 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.973065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.972951 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.973180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.973101 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.973397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.973372 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.973583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.973565 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.974862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.974839 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:36.975179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:36.975163 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:37.001065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.001031 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgdn6\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:37.001187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.001102 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\") pod \"e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:37.023177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.023155 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:37.371916 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:43:37.371884 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416812c2_aea7_4a8b_9d51_120ad7e9cf0f.slice/crio-1d4ba605e95a2c9eb101def39e299fdc2a4fe13e49308c140a1006e08383c91f WatchSource:0}: Error finding container 1d4ba605e95a2c9eb101def39e299fdc2a4fe13e49308c140a1006e08383c91f: Status 404 returned error can't find the container with id 1d4ba605e95a2c9eb101def39e299fdc2a4fe13e49308c140a1006e08383c91f Oct 30 17:43:37.372929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.372908 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:43:37.375603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.375570 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:37.375699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.375651 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:37.375699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.375681 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:37.483231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.483190 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" event={"ID":"416812c2-aea7-4a8b-9d51-120ad7e9cf0f","Type":"ContainerStarted","Data":"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c"} Oct 30 17:43:37.483643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.483252 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" event={"ID":"416812c2-aea7-4a8b-9d51-120ad7e9cf0f","Type":"ContainerStarted","Data":"1d4ba605e95a2c9eb101def39e299fdc2a4fe13e49308c140a1006e08383c91f"} Oct 30 17:43:37.531371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.531323 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podStartSLOduration=1.5313079520000001 podStartE2EDuration="1.531307952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:37.52867524 +0000 UTC m=+1845.306147197" watchObservedRunningTime="2026-10-30 17:43:37.531307952 +0000 UTC m=+1845.308779906" Oct 30 17:43:37.629542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.629460 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:37.629542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:37.629520 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:38.023987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:38.023903 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:38.025620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:38.025596 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:38.025749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:38.025642 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:38.630801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:38.630754 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:38.631438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:38.630826 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:39.023742 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:39.023646 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:39.024028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:39.023987 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:39.629759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:39.629722 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:39.629917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:39.629778 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:40.023860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:40.023774 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:40.023860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:40.023824 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:40.426806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:40.426773 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:43:40.629691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:40.629657 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:40.629885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:40.629709 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:41.057782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:41.057730 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:41.070985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:41.070942 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:41.629806 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:41.629766 4622 patch_prober.go:29] interesting pod/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:43:41.630025 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:41.629821 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 17:43:42.024227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:42.023687 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:42.024227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:42.023750 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:42.645602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:42.645569 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:43:42.656483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:42.656457 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:43:43.024357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:43.024243 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:43.024357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:43.024327 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:44.031872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.031362 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 17:43:44.031872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.031439 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 17:43:44.109299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.109251 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:43:44.113963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.113938 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.196286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.195191 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:43:44.224851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.224822 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24fb0d-openshift-default-dockercfg-wm569\"" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305622 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305724 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305756 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305787 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305838 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305892 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305940 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.305983 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.306334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.306090 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2mb6\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.407800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.407758 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.407988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.407834 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.407988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.407893 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408032 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2mb6\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408327 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408371 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408428 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408523 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408564 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408567 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408631 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408726 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408765 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408728 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.408904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.408793 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.411394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.411366 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.411540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.411520 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.432718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.432628 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2mb6\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.433352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.433332 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\") pod \"e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.520603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.520565 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:44.882536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.882508 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:43:44.883186 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:43:44.883158 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a116870_e9f7_41d5_98cc_828575727fd8.slice/crio-96c323ca67bfb41034e02d608bf623cf5ddb71186fb25a9cb973db0472880e23 WatchSource:0}: Error finding container 96c323ca67bfb41034e02d608bf623cf5ddb71186fb25a9cb973db0472880e23: Status 404 returned error can't find the container with id 96c323ca67bfb41034e02d608bf623cf5ddb71186fb25a9cb973db0472880e23 Oct 30 17:43:44.886601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.886565 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:44.886689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.886635 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:44.886737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:44.886666 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:43:45.030396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.030360 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:45.031118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.030884 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:45.032017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.031996 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:43:45.714519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.714483 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerStarted","Data":"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d"} Oct 30 17:43:45.714715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.714535 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerStarted","Data":"96c323ca67bfb41034e02d608bf623cf5ddb71186fb25a9cb973db0472880e23"} Oct 30 17:43:45.774565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:45.774510 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podStartSLOduration=1.7744963280000001 podStartE2EDuration="1.774496328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:45.773576225 +0000 UTC m=+1853.551048179" watchObservedRunningTime="2026-10-30 17:43:45.774496328 +0000 UTC m=+1853.551968283" Oct 30 17:43:46.521003 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:46.520969 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:46.526584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:46.526562 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:46.727399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:46.727369 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:46.728071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:43:46.728049 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:43:54.535334 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:43:54.535292 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:44:01.923148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:01.923119 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:44:01.928383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:01.928366 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:01.951300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:01.951274 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b1c345-serving-cert\"" Oct 30 17:44:01.958386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:01.958357 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:44:02.015753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.015723 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.015916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.015900 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.015963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.015931 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.016018 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.015959 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlw2\" (UniqueName: \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.121803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.121764 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.122001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.121825 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.122001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.121868 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlw2\" (UniqueName: \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.122370 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.122344 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.125405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.123923 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.153689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.153656 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.153968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.153893 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.154553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.154520 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlw2\" (UniqueName: \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\") pod \"maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.242699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.242612 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:02.545301 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:44:02.545237 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c829eb5_e40d_493a_bca0_4f8232debac7.slice/crio-58bf910bb9516ed675e16a657252a0bb662580982d05235dc1565cfc1b3b17a1 WatchSource:0}: Error finding container 58bf910bb9516ed675e16a657252a0bb662580982d05235dc1565cfc1b3b17a1: Status 404 returned error can't find the container with id 58bf910bb9516ed675e16a657252a0bb662580982d05235dc1565cfc1b3b17a1 Oct 30 17:44:02.545938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.545917 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:44:02.886673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:02.886024 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" event={"ID":"2c829eb5-e40d-493a-bca0-4f8232debac7","Type":"ContainerStarted","Data":"58bf910bb9516ed675e16a657252a0bb662580982d05235dc1565cfc1b3b17a1"} Oct 30 17:44:03.895233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:03.895199 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" event={"ID":"2c829eb5-e40d-493a-bca0-4f8232debac7","Type":"ContainerStarted","Data":"ee203e857a6e870ab3a3f159f6845e6ebe083698498d2ee39b20726da6053891"} Oct 30 17:44:03.895659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:03.895333 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:03.941153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:03.941098 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" podStartSLOduration=2.6033642 podStartE2EDuration="2.941079617s" totalImagesPullingTime="337.715417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:03.939438855 +0000 UTC m=+1871.716910811" watchObservedRunningTime="2026-10-30 17:44:03.941079617 +0000 UTC m=+1871.718551572" Oct 30 17:44:09.909083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:09.909052 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:44:24.534802 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:24.534762 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:44:24.610853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:24.610814 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:44:24.611313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:24.611059 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" podUID="118c0be6-649b-4f1f-bc67-4deca469ebb1" containerName="maas-api" containerID="cri-o://d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c" gracePeriod=30 Oct 30 17:44:25.106781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.105468 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:44:25.112287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.111997 4622 generic.go:356] "Generic (PLEG): container finished" podID="118c0be6-649b-4f1f-bc67-4deca469ebb1" containerID="d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c" exitCode=0 Oct 30 17:44:25.112287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.112173 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" event={"ID":"118c0be6-649b-4f1f-bc67-4deca469ebb1","Type":"ContainerDied","Data":"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c"} Oct 30 17:44:25.112287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.112227 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" event={"ID":"118c0be6-649b-4f1f-bc67-4deca469ebb1","Type":"ContainerDied","Data":"f8293a608233f787e55ece02626e8c5fcb3e0ce5afb4f4dc1ee4e1a767677251"} Oct 30 17:44:25.116559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.112256 4622 scope.go:122] "RemoveContainer" containerID="d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c" Oct 30 17:44:25.138293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.135997 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\") pod \"118c0be6-649b-4f1f-bc67-4deca469ebb1\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " Oct 30 17:44:25.138293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.136055 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\" (UniqueName: \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\") pod \"118c0be6-649b-4f1f-bc67-4deca469ebb1\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " Oct 30 17:44:25.138293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.136114 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\") pod \"118c0be6-649b-4f1f-bc67-4deca469ebb1\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " Oct 30 17:44:25.138293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.136286 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\") pod \"118c0be6-649b-4f1f-bc67-4deca469ebb1\" (UID: \"118c0be6-649b-4f1f-bc67-4deca469ebb1\") " Oct 30 17:44:25.138293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.137620 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca" pod "118c0be6-649b-4f1f-bc67-4deca469ebb1" (UID: "118c0be6-649b-4f1f-bc67-4deca469ebb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:25.154354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.152358 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls" pod "118c0be6-649b-4f1f-bc67-4deca469ebb1" (UID: "118c0be6-649b-4f1f-bc67-4deca469ebb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:44:25.169337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.163493 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs" pod "118c0be6-649b-4f1f-bc67-4deca469ebb1" (UID: "118c0be6-649b-4f1f-bc67-4deca469ebb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:44:25.169788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.169761 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84" pod "118c0be6-649b-4f1f-bc67-4deca469ebb1" (UID: "118c0be6-649b-4f1f-bc67-4deca469ebb1"). InnerVolumeSpecName "kube-api-access-vrj84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:25.239978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.239528 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrj84\" (UniqueName: \"kubernetes.io/projected/118c0be6-649b-4f1f-bc67-4deca469ebb1-kube-api-access-vrj84\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:25.239978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.239562 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:25.239978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.239594 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/118c0be6-649b-4f1f-bc67-4deca469ebb1-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:25.239978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.239608 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118c0be6-649b-4f1f-bc67-4deca469ebb1-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:25.240255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.239984 4622 scope.go:122] "RemoveContainer" containerID="d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c" Oct 30 17:44:25.242484 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:25.241047 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c\": container with ID starting with d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c not found: ID does not exist" containerID="d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c" Oct 30 17:44:25.242484 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.241091 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c"} err="failed to get container status \"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c\": rpc error: code = NotFound desc = could not find container \"d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c\": container with ID starting with d4598f2007ce4d55eac8ef5aaaac23930c280c2d7307344d316e24acfe50459c not found: ID does not exist" Oct 30 17:44:25.428213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.428132 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:44:25.428416 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.428381 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" containerID="cri-o://ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc" gracePeriod=30 Oct 30 17:44:25.678374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.678295 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:44:25.678758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.678549 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" containerID="cri-o://129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be" gracePeriod=30 Oct 30 17:44:25.682384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.682358 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:44:25.682619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.682571 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" containerID="cri-o://d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a" gracePeriod=30 Oct 30 17:44:25.895099 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.895066 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:44:25.895365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:25.895311 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" containerID="cri-o://f18e5526fb557f68dfef73f4a0dcb74df6e24991a975a320f1aec31f425c02a2" gracePeriod=30 Oct 30 17:44:26.129538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:26.129509 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck" Oct 30 17:44:26.185111 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:26.185079 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:44:26.188862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:26.188843 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49e155-5688bf9b6d-sf4ck"] Oct 30 17:44:26.375124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:26.375092 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118c0be6-649b-4f1f-bc67-4deca469ebb1" path="/var/lib/kubelet/pods/118c0be6-649b-4f1f-bc67-4deca469ebb1/volumes" Oct 30 17:44:28.215909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.215877 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:44:28.216561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.216547 4622 cpu_manager.go:414] "containerMap: removing container" podUID="118c0be6-649b-4f1f-bc67-4deca469ebb1" containerName="maas-api" Oct 30 17:44:28.216614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.216563 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118c0be6-649b-4f1f-bc67-4deca469ebb1" containerName="maas-api" Oct 30 17:44:28.216690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.216679 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="118c0be6-649b-4f1f-bc67-4deca469ebb1" containerName="maas-api" Oct 30 17:44:28.221767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.221747 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.240452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.240428 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:44:28.284897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.284863 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcjc\" (UniqueName: \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.285090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.285069 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.285160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.285107 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.285333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.285305 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.315452 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.315420 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ee9ff3-serving-cert\"" Oct 30 17:44:28.386419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.386392 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcjc\" (UniqueName: \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.386653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.386630 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.386815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.386763 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.387031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.387004 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.387518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.387492 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.389394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.389374 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.390085 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.390067 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.410084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.410061 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcjc\" (UniqueName: \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\") pod \"maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.535474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.535447 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:28.866143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:28.863479 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:44:28.866248 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:44:28.866208 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d01767_d84c_4869_81b5_b0581bfb4de1.slice/crio-453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff WatchSource:0}: Error finding container 453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff: Status 404 returned error can't find the container with id 453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff Oct 30 17:44:29.156153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:29.156061 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" event={"ID":"16d01767-d84c-4869-81b5-b0581bfb4de1","Type":"ContainerStarted","Data":"453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff"} Oct 30 17:44:30.165708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.165666 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" event={"ID":"16d01767-d84c-4869-81b5-b0581bfb4de1","Type":"ContainerStarted","Data":"ded6f404d3d76e957c31cc0950ab41de6b3c96a7dc9027bbbc60f78b374421b9"} Oct 30 17:44:30.166097 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.165751 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:30.208117 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.208067 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" podStartSLOduration=1.863996041 podStartE2EDuration="2.208056279s" totalImagesPullingTime="344.060238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:30.207569305 +0000 UTC m=+1897.985041259" watchObservedRunningTime="2026-10-30 17:44:30.208056279 +0000 UTC m=+1897.985528235" Oct 30 17:44:30.869483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.867920 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:44:30.929196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929167 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929240 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929398 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929479 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929499 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:30.929555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929549 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929631 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929693 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929754 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929795 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\") pod \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\" (UID: \"78329738-7ff8-4f4f-9b71-51cc9abc18cb\") " Oct 30 17:44:30.929826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.929817 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:30.930050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.930029 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.930496 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.930762 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.932859 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.932881 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.932894 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.932906 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.932918 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:30.934959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.933338 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:30.946346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.937696 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:30.946346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.938520 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:30.956229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:30.956200 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq" pod "78329738-7ff8-4f4f-9b71-51cc9abc18cb" (UID: "78329738-7ff8-4f4f-9b71-51cc9abc18cb"). InnerVolumeSpecName "kube-api-access-k9vvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.018524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.018498 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:44:31.023938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.023919 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:44:31.034139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.034116 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.034256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.034141 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vvq\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-kube-api-access-k9vvq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.034256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.034159 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.034256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.034169 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78329738-7ff8-4f4f-9b71-51cc9abc18cb-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.135461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135423 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.135598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135495 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.135598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135581 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.135734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135610 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.135734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135645 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.135734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135686 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.135895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135735 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.135895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135736 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.135895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135803 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.135895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135856 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.135895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135813 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135918 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135958 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.135992 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136030 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136070 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.136167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136140 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.136500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136162 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.136622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136506 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.136622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136522 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.136622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136593 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\") pod \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\" (UID: \"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f\") " Oct 30 17:44:31.136780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136711 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.136842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.136800 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\") pod \"479bdec4-2eb5-450f-ab89-e4c903f001a6\" (UID: \"479bdec4-2eb5-450f-ab89-e4c903f001a6\") " Oct 30 17:44:31.137119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137095 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.137444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137419 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.137957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137935 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137966 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137980 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.137996 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479bdec4-2eb5-450f-ab89-e4c903f001a6-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.138007 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.138017 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.138030 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.138483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.138458 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.138866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.138840 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.139437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.139092 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.139437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.139368 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.139651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.139627 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.139753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.139734 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "kube-api-access-lprcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.139885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.139865 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.140660 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.140636 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.140950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.140925 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo" pod "479bdec4-2eb5-450f-ab89-e4c903f001a6" (UID: "479bdec4-2eb5-450f-ab89-e4c903f001a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.141142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.141122 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "kube-api-access-zh7f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.141543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.141516 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token" pod "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" (UID: "69a9d4d4-0e3e-4524-98a2-e62ac8265d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.174949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.174923 4622 generic.go:356] "Generic (PLEG): container finished" podID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerID="129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be" exitCode=0 Oct 30 17:44:31.175382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.174965 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" Oct 30 17:44:31.175382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.175000 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" event={"ID":"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f","Type":"ContainerDied","Data":"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be"} Oct 30 17:44:31.175382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.175049 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q" event={"ID":"69a9d4d4-0e3e-4524-98a2-e62ac8265d5f","Type":"ContainerDied","Data":"f1ba53d2ebc7f5ee0eb78719ade51ed41c9266618133a3ba9011183874f9b010"} Oct 30 17:44:31.175382 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.175070 4622 scope.go:122] "RemoveContainer" containerID="129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be" Oct 30 17:44:31.176692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.176669 4622 generic.go:356] "Generic (PLEG): container finished" podID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerID="d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a" exitCode=0 Oct 30 17:44:31.176790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.176729 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" Oct 30 17:44:31.176790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.176750 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" event={"ID":"479bdec4-2eb5-450f-ab89-e4c903f001a6","Type":"ContainerDied","Data":"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a"} Oct 30 17:44:31.176919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.176795 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk" event={"ID":"479bdec4-2eb5-450f-ab89-e4c903f001a6","Type":"ContainerDied","Data":"5b85277b998ae2f756f978a5fcfd455ea74037a4499b818cd096f6a86756923f"} Oct 30 17:44:31.181740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.181711 4622 generic.go:356] "Generic (PLEG): container finished" podID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerID="ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc" exitCode=0 Oct 30 17:44:31.181848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.181824 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" event={"ID":"78329738-7ff8-4f4f-9b71-51cc9abc18cb","Type":"ContainerDied","Data":"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc"} Oct 30 17:44:31.181848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.181835 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" Oct 30 17:44:31.181997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.181861 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql" event={"ID":"78329738-7ff8-4f4f-9b71-51cc9abc18cb","Type":"ContainerDied","Data":"3fc31412853a4851982f7f61adfeccee3a718deef2f53138a14a2a6734360420"} Oct 30 17:44:31.184498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.183994 4622 generic.go:356] "Generic (PLEG): container finished" podID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerID="f18e5526fb557f68dfef73f4a0dcb74df6e24991a975a320f1aec31f425c02a2" exitCode=0 Oct 30 17:44:31.184498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.184096 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" event={"ID":"41ca7223-c3d5-40bb-be56-b7b4858e90df","Type":"ContainerDied","Data":"f18e5526fb557f68dfef73f4a0dcb74df6e24991a975a320f1aec31f425c02a2"} Oct 30 17:44:31.195352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.195334 4622 scope.go:122] "RemoveContainer" containerID="129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be" Oct 30 17:44:31.196105 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:31.196082 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be\": container with ID starting with 129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be not found: ID does not exist" containerID="129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be" Oct 30 17:44:31.196199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.196118 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be"} err="failed to get container status \"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be\": rpc error: code = NotFound desc = could not find container \"129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be\": container with ID starting with 129f31a7396a82ea5ba2db2fd13f6c42fc6efa2846021eee7f9b29334c76f3be not found: ID does not exist" Oct 30 17:44:31.196199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.196136 4622 scope.go:122] "RemoveContainer" containerID="d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a" Oct 30 17:44:31.204220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.204203 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:44:31.206229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.206213 4622 scope.go:122] "RemoveContainer" containerID="d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a" Oct 30 17:44:31.206541 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:31.206524 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a\": container with ID starting with d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a not found: ID does not exist" containerID="d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a" Oct 30 17:44:31.206610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.206551 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a"} err="failed to get container status \"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a\": rpc error: code = NotFound desc = could not find container \"d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a\": container with ID starting with d742257d70f9ff3d8a97b5a37aa6c50aa5e945dbaadcb31de062da530c50e89a not found: ID does not exist" Oct 30 17:44:31.206610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.206572 4622 scope.go:122] "RemoveContainer" containerID="ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc" Oct 30 17:44:31.215216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.215200 4622 scope.go:122] "RemoveContainer" containerID="ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc" Oct 30 17:44:31.215587 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:31.215566 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc\": container with ID starting with ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc not found: ID does not exist" containerID="ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc" Oct 30 17:44:31.215648 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.215599 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc"} err="failed to get container status \"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc\": rpc error: code = NotFound desc = could not find container \"ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc\": container with ID starting with ffe475928798bb96a9f89daa368ca05f6d93dbe2e7e7db1bb59d0bdc1fa471bc not found: ID does not exist" Oct 30 17:44:31.237617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.237585 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:44:31.239297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239251 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239355 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239389 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239489 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239521 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.239643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239585 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239642 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239701 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239730 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.239857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239776 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.239857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.239829 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\") pod \"41ca7223-c3d5-40bb-be56-b7b4858e90df\" (UID: \"41ca7223-c3d5-40bb-be56-b7b4858e90df\") " Oct 30 17:44:31.240714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240690 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240716 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240730 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240743 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240755 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240768 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240781 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240793 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240806 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7f6\" (UniqueName: \"kubernetes.io/projected/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-kube-api-access-zh7f6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240818 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479bdec4-2eb5-450f-ab89-e4c903f001a6-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240832 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lprcj\" (UniqueName: \"kubernetes.io/projected/479bdec4-2eb5-450f-ab89-e4c903f001a6-kube-api-access-lprcj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240845 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.240881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.240859 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.241501 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.241305 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.241665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.241646 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:31.241858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.241835 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:31.241957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.241928 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.242164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.242144 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.242164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.242155 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "kube-api-access-5hgjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:31.242859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.242840 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-w4p2q"] Oct 30 17:44:31.243457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.243438 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy" pod "41ca7223-c3d5-40bb-be56-b7b4858e90df" (UID: "41ca7223-c3d5-40bb-be56-b7b4858e90df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:31.302538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.302514 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:44:31.306242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.306222 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-7b5sk"] Oct 30 17:44:31.343457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343428 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343459 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343474 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hgjj\" (UniqueName: \"kubernetes.io/projected/41ca7223-c3d5-40bb-be56-b7b4858e90df-kube-api-access-5hgjj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343489 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ca7223-c3d5-40bb-be56-b7b4858e90df-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343502 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343515 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.343658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.343527 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ca7223-c3d5-40bb-be56-b7b4858e90df-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:31.403771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.401345 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:44:31.428053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:31.427192 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-z4kql"] Oct 30 17:44:32.197816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.197730 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" Oct 30 17:44:32.197816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.197753 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh" event={"ID":"41ca7223-c3d5-40bb-be56-b7b4858e90df","Type":"ContainerDied","Data":"f2c4b935ff388568ed913daaf3a9fc3c9d177dd324a77fd612be781aef9f990d"} Oct 30 17:44:32.198372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.197819 4622 scope.go:122] "RemoveContainer" containerID="f18e5526fb557f68dfef73f4a0dcb74df6e24991a975a320f1aec31f425c02a2" Oct 30 17:44:32.276352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.276318 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:44:32.279803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.279776 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-f94vh"] Oct 30 17:44:32.379845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.378833 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" path="/var/lib/kubelet/pods/41ca7223-c3d5-40bb-be56-b7b4858e90df/volumes" Oct 30 17:44:32.379845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.379562 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" path="/var/lib/kubelet/pods/479bdec4-2eb5-450f-ab89-e4c903f001a6/volumes" Oct 30 17:44:32.380314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.380290 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" path="/var/lib/kubelet/pods/69a9d4d4-0e3e-4524-98a2-e62ac8265d5f/volumes" Oct 30 17:44:32.381573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:32.381550 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" path="/var/lib/kubelet/pods/78329738-7ff8-4f4f-9b71-51cc9abc18cb/volumes" Oct 30 17:44:34.465410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:34.465378 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:44:34.465823 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:34.465669 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" containerID="cri-o://1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a" gracePeriod=30 Oct 30 17:44:35.802068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.802030 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:44:35.803108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803080 4622 cpu_manager.go:414] "containerMap: removing container" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" Oct 30 17:44:35.803108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803109 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803130 4622 cpu_manager.go:414] "containerMap: removing container" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803138 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803161 4622 cpu_manager.go:414] "containerMap: removing container" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803169 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803219 4622 cpu_manager.go:414] "containerMap: removing container" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" Oct 30 17:44:35.803369 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803228 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" Oct 30 17:44:35.803681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803390 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a9d4d4-0e3e-4524-98a2-e62ac8265d5f" containerName="istio-proxy" Oct 30 17:44:35.803681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803413 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="78329738-7ff8-4f4f-9b71-51cc9abc18cb" containerName="istio-proxy" Oct 30 17:44:35.803681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803426 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="479bdec4-2eb5-450f-ab89-e4c903f001a6" containerName="istio-proxy" Oct 30 17:44:35.803681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.803438 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ca7223-c3d5-40bb-be56-b7b4858e90df" containerName="istio-proxy" Oct 30 17:44:35.807473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.807450 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.828939 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.828919 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-488c67-openshift-default-dockercfg-2l4gz\"" Oct 30 17:44:35.847138 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.847113 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929201 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm7b\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929306 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929365 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929409 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929449 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929492 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929555 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929614 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:35.933287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:35.929723 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032583 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm7b\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032681 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032753 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032815 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032869 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032914 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.032996 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.033064 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.033306 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.041287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.039921 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.052065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.046881 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.052065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.047152 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.052065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.047392 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.052065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.047833 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.053422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.053334 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.064019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.063962 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm7b\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.068315 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.064502 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.098285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.087818 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\") pod \"e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.126852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.126545 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:36.191998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.191960 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:44:36.508597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.508566 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:44:36.509155 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:44:36.509126 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8df3750_cc02_42da_8e32_694298a39b89.slice/crio-4e96848163776d858943de9ea440de8039d84590065f2e73abf7288f1bcdd699 WatchSource:0}: Error finding container 4e96848163776d858943de9ea440de8039d84590065f2e73abf7288f1bcdd699: Status 404 returned error can't find the container with id 4e96848163776d858943de9ea440de8039d84590065f2e73abf7288f1bcdd699 Oct 30 17:44:36.512726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.512689 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:44:36.512845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.512775 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:44:36.512845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:36.512819 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:44:37.304160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:37.304080 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerStarted","Data":"5694eb7dbd5af0866d7d2134c3892e7c2ff006932573b59a31853a8a7f532dd0"} Oct 30 17:44:37.304160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:37.304162 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerStarted","Data":"4e96848163776d858943de9ea440de8039d84590065f2e73abf7288f1bcdd699"} Oct 30 17:44:37.377930 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:37.377880 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podStartSLOduration=2.377865361 podStartE2EDuration="2.377865361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:44:37.361840622 +0000 UTC m=+1905.139312582" watchObservedRunningTime="2026-10-30 17:44:37.377865361 +0000 UTC m=+1905.155337316" Oct 30 17:44:38.127965 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.127918 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:38.135635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.135604 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:38.314172 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.314141 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:38.315275 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.315237 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:44:38.980287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.976588 4622 patch_prober.go:29] interesting pod/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:44:38.980287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:38.976677 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:44:39.753244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.753221 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:44:39.839770 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.839741 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.839954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.839906 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.839954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.839945 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840035 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840069 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840143 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840281 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:39.840395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840186 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:44:39.840395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840285 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840386 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840453 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\") pod \"0a90956a-5776-4e90-85c8-0a3bdd46884f\" (UID: \"0a90956a-5776-4e90-85c8-0a3bdd46884f\") " Oct 30 17:44:39.840557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840502 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:39.840557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840516 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:39.840722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.840705 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:39.841166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.841142 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.841317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.841170 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.841317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.841184 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.841317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.841195 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.841317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.841207 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90956a-5776-4e90-85c8-0a3bdd46884f-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.842959 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.842921 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:39.843130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.843109 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "kube-api-access-6tlzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:44:39.843462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.843435 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:44:39.843577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.843484 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo" pod "0a90956a-5776-4e90-85c8-0a3bdd46884f" (UID: "0a90956a-5776-4e90-85c8-0a3bdd46884f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:44:39.942718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.942683 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.942718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.942711 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.942718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.942720 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90956a-5776-4e90-85c8-0a3bdd46884f-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:39.942718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:39.942728 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tlzr\" (UniqueName: \"kubernetes.io/projected/0a90956a-5776-4e90-85c8-0a3bdd46884f-kube-api-access-6tlzr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:44:40.334573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.334537 4622 generic.go:356] "Generic (PLEG): container finished" podID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerID="1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a" exitCode=0 Oct 30 17:44:40.334747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.334591 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" Oct 30 17:44:40.334747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.334616 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerDied","Data":"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a"} Oct 30 17:44:40.334747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.334674 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l" event={"ID":"0a90956a-5776-4e90-85c8-0a3bdd46884f","Type":"ContainerDied","Data":"c3278bf8aafa7d9a2a0bf78a76704faef956e33b07be44efbc460f5b2e1354c1"} Oct 30 17:44:40.334747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.334702 4622 scope.go:122] "RemoveContainer" containerID="1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a" Oct 30 17:44:40.347140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.347123 4622 scope.go:122] "RemoveContainer" containerID="2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4" Oct 30 17:44:40.360211 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.360189 4622 scope.go:122] "RemoveContainer" containerID="1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a" Oct 30 17:44:40.360528 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:40.360507 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a\": container with ID starting with 1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a not found: ID does not exist" containerID="1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a" Oct 30 17:44:40.360623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.360539 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a"} err="failed to get container status \"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a\": rpc error: code = NotFound desc = could not find container \"1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a\": container with ID starting with 1151089c030ecdec79fca859ae0c3cc89635787929292d88ac564da00392122a not found: ID does not exist" Oct 30 17:44:40.360623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.360557 4622 scope.go:122] "RemoveContainer" containerID="2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4" Oct 30 17:44:40.360859 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:40.360839 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4\": container with ID starting with 2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4 not found: ID does not exist" containerID="2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4" Oct 30 17:44:40.360908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.360866 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4"} err="failed to get container status \"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4\": rpc error: code = NotFound desc = could not find container \"2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4\": container with ID starting with 2b240f3a6b1acbc05403bd8f10127400fa3c134bd6644b3d9cfde3d85fb4d9b4 not found: ID does not exist" Oct 30 17:44:40.403763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.403740 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:44:40.418394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:40.417871 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-49e155-openshift-default-765d8b6c89-htj7l"] Oct 30 17:44:42.375907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:44:42.375869 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" path="/var/lib/kubelet/pods/0a90956a-5776-4e90-85c8-0a3bdd46884f/volumes" Oct 30 17:44:54.535249 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:44:54.535211 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:45:00.176391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.176350 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:00.177568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.177541 4622 cpu_manager.go:414] "containerMap: removing container" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.177756 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.177725 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.177974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.177956 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.178139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.178109 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.186503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.186481 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:00.186700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.186674 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.187213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.187193 4622 cpu_manager.go:414] "containerMap: removing container" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.187213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.187213 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a90956a-5776-4e90-85c8-0a3bdd46884f" containerName="istio-proxy" Oct 30 17:45:00.193414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.193390 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:00.193657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.193634 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.198486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.198468 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.208841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.208818 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:45:00.215803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.215740 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:00.235512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.235424 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:00.247334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.247312 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:00.247474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.247358 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:00.247474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.247381 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.247843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.247821 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:00.251535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.251512 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:00.257489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.257469 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:00.257724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.257572 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.262925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.262904 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.269784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.269759 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:00.279449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.279432 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.285684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.285643 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcb8s\" (UniqueName: \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.286172 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.286154 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.295107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.295084 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:00.304280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.301296 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:00.344584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.344558 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:00.388239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388201 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcb8s\" (UniqueName: \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.388440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388251 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.388440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388331 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.388440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388386 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwh6\" (UniqueName: \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.388440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388421 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.388669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388449 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxdj\" (UniqueName: \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.388669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388593 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.388780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388679 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.388780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388722 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf59z\" (UniqueName: \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.388887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388758 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgc5\" (UniqueName: \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.388887 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.388812 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7w6\" (UniqueName: \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.389475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.389296 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.389475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.389378 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fsg\" (UniqueName: \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.389475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.389414 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.390228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.390158 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.412387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.412358 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcb8s\" (UniqueName: \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\") pod \"maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.491157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491087 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.491157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491132 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf59z\" (UniqueName: \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.491157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491155 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgc5\" (UniqueName: \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.491437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491177 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7w6\" (UniqueName: \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.491437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491238 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fsg\" (UniqueName: \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.491437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491294 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.491437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491382 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.491657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491463 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.491657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491536 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwh6\" (UniqueName: \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.491657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491573 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.491657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491591 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxdj\" (UniqueName: \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.491874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491736 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.491874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.491844 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.492195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.492169 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.492368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.492348 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.492463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.492423 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.492578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.492555 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.492792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.492775 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.512475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.512454 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:00.515203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.515181 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwh6\" (UniqueName: \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\") pod \"maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.515310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.515181 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf59z\" (UniqueName: \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\") pod \"maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.515425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.515401 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgc5\" (UniqueName: \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\") pod \"maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.515495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.515481 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxdj\" (UniqueName: \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\") pod \"maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.515540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.515491 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fsg\" (UniqueName: \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\") pod \"maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.516068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.516045 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7w6\" (UniqueName: \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\") pod \"maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.538094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.538073 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:00.557224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.557201 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:00.579938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.579473 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:00.587852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.587828 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:00.632412 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.629216 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:00.647256 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.645079 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:00.829415 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:00.829382 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb69423_836c_4e78_805b_fea52bbf1127.slice/crio-10a790c740f573f7f32b8f75392c22280dbe90e95b393e545b9e2ac34f46bf98 WatchSource:0}: Error finding container 10a790c740f573f7f32b8f75392c22280dbe90e95b393e545b9e2ac34f46bf98: Status 404 returned error can't find the container with id 10a790c740f573f7f32b8f75392c22280dbe90e95b393e545b9e2ac34f46bf98 Oct 30 17:45:00.830804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.830773 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:00.862702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.862680 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:00.996868 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:00.996839 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b83092_530e_47fd_a57e_0c328da69269.slice/crio-ede01fbf8332cf06aafcf57c4c16896f3df833ff1808684be927572786f551f6 WatchSource:0}: Error finding container ede01fbf8332cf06aafcf57c4c16896f3df833ff1808684be927572786f551f6: Status 404 returned error can't find the container with id ede01fbf8332cf06aafcf57c4c16896f3df833ff1808684be927572786f551f6 Oct 30 17:45:00.998011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:00.997992 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:01.002519 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:01.002482 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f010a2_2a45_41f9_af45_9a9483de0032.slice/crio-f9a13356a05db24c302633fea37efe6c17bbe5ab6551e166beb68b10629fb068 WatchSource:0}: Error finding container f9a13356a05db24c302633fea37efe6c17bbe5ab6551e166beb68b10629fb068: Status 404 returned error can't find the container with id f9a13356a05db24c302633fea37efe6c17bbe5ab6551e166beb68b10629fb068 Oct 30 17:45:01.003736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.003707 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:01.008407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.008386 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:01.011732 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:01.011708 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b137c19_15e4_4461_ac93_03cce05a72f1.slice/crio-8ef7967699e8f8b671dd9e74e8aef293785b07606e9bd7e1ad095cf96e91c531 WatchSource:0}: Error finding container 8ef7967699e8f8b671dd9e74e8aef293785b07606e9bd7e1ad095cf96e91c531: Status 404 returned error can't find the container with id 8ef7967699e8f8b671dd9e74e8aef293785b07606e9bd7e1ad095cf96e91c531 Oct 30 17:45:01.101780 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:01.101751 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38439bc_b82d_4a24_95fd_9c9041c8b5f8.slice/crio-aa84d57c52cd5435336320fb1c59ec3f391c6c00364689940dde06f501c7228c WatchSource:0}: Error finding container aa84d57c52cd5435336320fb1c59ec3f391c6c00364689940dde06f501c7228c: Status 404 returned error can't find the container with id aa84d57c52cd5435336320fb1c59ec3f391c6c00364689940dde06f501c7228c Oct 30 17:45:01.102627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.102598 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:01.146915 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:01.146883 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd26ed1_f9ee_4753_9838_06d7e1c15808.slice/crio-a5b80c53236b9261fd5577754e3ea6d9f0165e189b7b2a4c30a29d6cccbcfb52 WatchSource:0}: Error finding container a5b80c53236b9261fd5577754e3ea6d9f0165e189b7b2a4c30a29d6cccbcfb52: Status 404 returned error can't find the container with id a5b80c53236b9261fd5577754e3ea6d9f0165e189b7b2a4c30a29d6cccbcfb52 Oct 30 17:45:01.148874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.148855 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:01.489324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.489292 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerStarted","Data":"bd4bbc956ee9c639b2d77e0cc7d2befff877dcc51159709b7c8c62992b4fe959"} Oct 30 17:45:01.489772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.489353 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerStarted","Data":"aa84d57c52cd5435336320fb1c59ec3f391c6c00364689940dde06f501c7228c"} Oct 30 17:45:01.490639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.490606 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerStarted","Data":"f9a13356a05db24c302633fea37efe6c17bbe5ab6551e166beb68b10629fb068"} Oct 30 17:45:01.492190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.492162 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerStarted","Data":"8ef7967699e8f8b671dd9e74e8aef293785b07606e9bd7e1ad095cf96e91c531"} Oct 30 17:45:01.493505 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.493478 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerStarted","Data":"a5b80c53236b9261fd5577754e3ea6d9f0165e189b7b2a4c30a29d6cccbcfb52"} Oct 30 17:45:01.495504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.495477 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerStarted","Data":"5063e70eded3bfd086cf28da8a8f1d54f423596257ea82e176f9b69101e5ff5f"} Oct 30 17:45:01.495646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.495526 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerStarted","Data":"ede01fbf8332cf06aafcf57c4c16896f3df833ff1808684be927572786f551f6"} Oct 30 17:45:01.497392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.497366 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerStarted","Data":"5c7801437d12b73f61f7d20efd30350bce4195e552e9e337a2b4220807f98e8a"} Oct 30 17:45:01.497518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.497421 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerStarted","Data":"66da2ea38555e11b1a55cd7d62585660ec36725d6079df5c2b32ac569385c516"} Oct 30 17:45:01.499084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.499063 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerStarted","Data":"69c43e3dc1b165628368fae00b582d4c045d7d65e903d989307eebddc49fd140"} Oct 30 17:45:01.499192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.499100 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerStarted","Data":"10a790c740f573f7f32b8f75392c22280dbe90e95b393e545b9e2ac34f46bf98"} Oct 30 17:45:01.552529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.552483 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" podStartSLOduration=1.552469292 podStartE2EDuration="1.552469292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:01.549612152 +0000 UTC m=+1929.327084107" watchObservedRunningTime="2026-10-30 17:45:01.552469292 +0000 UTC m=+1929.329941246" Oct 30 17:45:01.601141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.601085 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" podStartSLOduration=1.601072074 podStartE2EDuration="1.601072074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:01.599308813 +0000 UTC m=+1929.376780768" watchObservedRunningTime="2026-10-30 17:45:01.601072074 +0000 UTC m=+1929.378544029" Oct 30 17:45:01.641846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.641789 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" podStartSLOduration=1.641772285 podStartE2EDuration="1.641772285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:01.639445949 +0000 UTC m=+1929.416917904" watchObservedRunningTime="2026-10-30 17:45:01.641772285 +0000 UTC m=+1929.419244240" Oct 30 17:45:01.683088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:01.683037 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" podStartSLOduration=1.6830245750000001 podStartE2EDuration="1.683024575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:01.681240247 +0000 UTC m=+1929.458712203" watchObservedRunningTime="2026-10-30 17:45:01.683024575 +0000 UTC m=+1929.460496549" Oct 30 17:45:02.508772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.508740 4622 generic.go:356] "Generic (PLEG): container finished" podID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerID="bd4bbc956ee9c639b2d77e0cc7d2befff877dcc51159709b7c8c62992b4fe959" exitCode=60 Oct 30 17:45:02.509179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.508821 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerDied","Data":"bd4bbc956ee9c639b2d77e0cc7d2befff877dcc51159709b7c8c62992b4fe959"} Oct 30 17:45:02.509179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.509095 4622 scope.go:122] "RemoveContainer" containerID="bd4bbc956ee9c639b2d77e0cc7d2befff877dcc51159709b7c8c62992b4fe959" Oct 30 17:45:02.510608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.510587 4622 generic.go:356] "Generic (PLEG): container finished" podID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerID="dc6d47383e2066a97d6d7ad9460af93de33fe7cee7a985f8d92f6c8468d65882" exitCode=60 Oct 30 17:45:02.510731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.510649 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerDied","Data":"dc6d47383e2066a97d6d7ad9460af93de33fe7cee7a985f8d92f6c8468d65882"} Oct 30 17:45:02.510934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.510918 4622 scope.go:122] "RemoveContainer" containerID="dc6d47383e2066a97d6d7ad9460af93de33fe7cee7a985f8d92f6c8468d65882" Oct 30 17:45:02.512457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.512420 4622 generic.go:356] "Generic (PLEG): container finished" podID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerID="56339fa25efa76b51b9b68adf14bd6c50386138c2df36219a6b7ed4b1f1849be" exitCode=60 Oct 30 17:45:02.512687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.512500 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerDied","Data":"56339fa25efa76b51b9b68adf14bd6c50386138c2df36219a6b7ed4b1f1849be"} Oct 30 17:45:02.512848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.512742 4622 scope.go:122] "RemoveContainer" containerID="56339fa25efa76b51b9b68adf14bd6c50386138c2df36219a6b7ed4b1f1849be" Oct 30 17:45:02.514435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.514340 4622 generic.go:356] "Generic (PLEG): container finished" podID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerID="6cb251f8c382cee2d6399e98b6959464db05230d6d7e0e39852dbeb9f5e63bd3" exitCode=60 Oct 30 17:45:02.514493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.514425 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerDied","Data":"6cb251f8c382cee2d6399e98b6959464db05230d6d7e0e39852dbeb9f5e63bd3"} Oct 30 17:45:02.514609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.514594 4622 scope.go:122] "RemoveContainer" containerID="6cb251f8c382cee2d6399e98b6959464db05230d6d7e0e39852dbeb9f5e63bd3" Oct 30 17:45:02.516181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.516154 4622 generic.go:356] "Generic (PLEG): container finished" podID="a0b83092-530e-47fd-a57e-0c328da69269" containerID="5063e70eded3bfd086cf28da8a8f1d54f423596257ea82e176f9b69101e5ff5f" exitCode=60 Oct 30 17:45:02.516310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.516209 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerDied","Data":"5063e70eded3bfd086cf28da8a8f1d54f423596257ea82e176f9b69101e5ff5f"} Oct 30 17:45:02.516512 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.516495 4622 scope.go:122] "RemoveContainer" containerID="5063e70eded3bfd086cf28da8a8f1d54f423596257ea82e176f9b69101e5ff5f" Oct 30 17:45:02.517889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.517871 4622 generic.go:356] "Generic (PLEG): container finished" podID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerID="5c7801437d12b73f61f7d20efd30350bce4195e552e9e337a2b4220807f98e8a" exitCode=60 Oct 30 17:45:02.517987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.517939 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerDied","Data":"5c7801437d12b73f61f7d20efd30350bce4195e552e9e337a2b4220807f98e8a"} Oct 30 17:45:02.518181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.518162 4622 scope.go:122] "RemoveContainer" containerID="5c7801437d12b73f61f7d20efd30350bce4195e552e9e337a2b4220807f98e8a" Oct 30 17:45:02.519651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.519631 4622 generic.go:356] "Generic (PLEG): container finished" podID="2eb69423-836c-4e78-805b-fea52bbf1127" containerID="69c43e3dc1b165628368fae00b582d4c045d7d65e903d989307eebddc49fd140" exitCode=60 Oct 30 17:45:02.519737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.519682 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerDied","Data":"69c43e3dc1b165628368fae00b582d4c045d7d65e903d989307eebddc49fd140"} Oct 30 17:45:02.520058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:02.520042 4622 scope.go:122] "RemoveContainer" containerID="69c43e3dc1b165628368fae00b582d4c045d7d65e903d989307eebddc49fd140" Oct 30 17:45:03.530094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.530002 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerStarted","Data":"925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb"} Oct 30 17:45:03.532140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.532099 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerStarted","Data":"812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86"} Oct 30 17:45:03.534053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.534028 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerStarted","Data":"977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23"} Oct 30 17:45:03.535920 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.535894 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerStarted","Data":"5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896"} Oct 30 17:45:03.537931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.537907 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerStarted","Data":"f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8"} Oct 30 17:45:03.540093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.540070 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerStarted","Data":"c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058"} Oct 30 17:45:03.544307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.543080 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerStarted","Data":"137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441"} Oct 30 17:45:03.698569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.698516 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" podStartSLOduration=3.69849806 podStartE2EDuration="3.69849806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:03.696526294 +0000 UTC m=+1931.473998248" watchObservedRunningTime="2026-10-30 17:45:03.69849806 +0000 UTC m=+1931.475970019" Oct 30 17:45:03.738414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.738372 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" podStartSLOduration=3.738356918 podStartE2EDuration="3.738356918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:03.735960401 +0000 UTC m=+1931.513432356" watchObservedRunningTime="2026-10-30 17:45:03.738356918 +0000 UTC m=+1931.515828873" Oct 30 17:45:03.776847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:03.776801 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" podStartSLOduration=3.776784096 podStartE2EDuration="3.776784096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:03.774845025 +0000 UTC m=+1931.552316983" watchObservedRunningTime="2026-10-30 17:45:03.776784096 +0000 UTC m=+1931.554256050" Oct 30 17:45:03.987764 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:03.987732 4622 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb69423_836c_4e78_805b_fea52bbf1127.slice/crio-977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:45:04.557761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.557729 4622 generic.go:356] "Generic (PLEG): container finished" podID="a0b83092-530e-47fd-a57e-0c328da69269" containerID="925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb" exitCode=60 Oct 30 17:45:04.558295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.557809 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerDied","Data":"925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb"} Oct 30 17:45:04.558295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.557872 4622 scope.go:122] "RemoveContainer" containerID="5063e70eded3bfd086cf28da8a8f1d54f423596257ea82e176f9b69101e5ff5f" Oct 30 17:45:04.558295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.558095 4622 scope.go:122] "RemoveContainer" containerID="925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb" Oct 30 17:45:04.558543 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.558509 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl_odh-ai-gateway-infra(a0b83092-530e-47fd-a57e-0c328da69269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" podUID="a0b83092-530e-47fd-a57e-0c328da69269" Oct 30 17:45:04.559938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.559916 4622 generic.go:356] "Generic (PLEG): container finished" podID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerID="812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86" exitCode=60 Oct 30 17:45:04.560026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.559942 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerDied","Data":"812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86"} Oct 30 17:45:04.560237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.560225 4622 scope.go:122] "RemoveContainer" containerID="812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86" Oct 30 17:45:04.560583 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.560559 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn_odh-ai-gateway-infra(ba3befca-fc29-4461-a246-d1eff0234a7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" Oct 30 17:45:04.574349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.574323 4622 generic.go:356] "Generic (PLEG): container finished" podID="2eb69423-836c-4e78-805b-fea52bbf1127" containerID="977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23" exitCode=60 Oct 30 17:45:04.574843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.574598 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerDied","Data":"977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23"} Oct 30 17:45:04.574843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.574750 4622 scope.go:122] "RemoveContainer" containerID="977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23" Oct 30 17:45:04.575053 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.575029 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5_odh-ai-gateway-infra(2eb69423-836c-4e78-805b-fea52bbf1127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" Oct 30 17:45:04.577272 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.577236 4622 generic.go:356] "Generic (PLEG): container finished" podID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerID="5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896" exitCode=60 Oct 30 17:45:04.577730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.577283 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerDied","Data":"5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896"} Oct 30 17:45:04.577730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.577558 4622 scope.go:122] "RemoveContainer" containerID="5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896" Oct 30 17:45:04.577874 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.577801 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl_odh-ai-gateway-infra(d38439bc-b82d-4a24-95fd-9c9041c8b5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" Oct 30 17:45:04.579910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.579886 4622 generic.go:356] "Generic (PLEG): container finished" podID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerID="f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8" exitCode=60 Oct 30 17:45:04.579998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.579965 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerDied","Data":"f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8"} Oct 30 17:45:04.580156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.580141 4622 scope.go:122] "RemoveContainer" containerID="f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8" Oct 30 17:45:04.580417 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.580395 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf_odh-ai-gateway-infra(b0f010a2-2a45-41f9-af45-9a9483de0032)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" Oct 30 17:45:04.581906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.581882 4622 generic.go:356] "Generic (PLEG): container finished" podID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerID="c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058" exitCode=60 Oct 30 17:45:04.581999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.581908 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerDied","Data":"c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058"} Oct 30 17:45:04.582167 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.582150 4622 scope.go:122] "RemoveContainer" containerID="c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058" Oct 30 17:45:04.582435 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.582411 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx_odh-ai-gateway-infra(2b137c19-15e4-4461-ac93-03cce05a72f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" Oct 30 17:45:04.583790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.583770 4622 generic.go:356] "Generic (PLEG): container finished" podID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerID="137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441" exitCode=60 Oct 30 17:45:04.583889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.583852 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerDied","Data":"137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441"} Oct 30 17:45:04.584087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.584068 4622 scope.go:122] "RemoveContainer" containerID="137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441" Oct 30 17:45:04.584442 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:04.584420 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm_odh-ai-gateway-infra(acd26ed1-f9ee-4753-9838-06d7e1c15808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" Oct 30 17:45:04.584908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.584894 4622 scope.go:122] "RemoveContainer" containerID="5c7801437d12b73f61f7d20efd30350bce4195e552e9e337a2b4220807f98e8a" Oct 30 17:45:04.598851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.598823 4622 scope.go:122] "RemoveContainer" containerID="69c43e3dc1b165628368fae00b582d4c045d7d65e903d989307eebddc49fd140" Oct 30 17:45:04.614691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.614670 4622 scope.go:122] "RemoveContainer" containerID="bd4bbc956ee9c639b2d77e0cc7d2befff877dcc51159709b7c8c62992b4fe959" Oct 30 17:45:04.628838 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.628819 4622 scope.go:122] "RemoveContainer" containerID="dc6d47383e2066a97d6d7ad9460af93de33fe7cee7a985f8d92f6c8468d65882" Oct 30 17:45:04.644862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.644843 4622 scope.go:122] "RemoveContainer" containerID="56339fa25efa76b51b9b68adf14bd6c50386138c2df36219a6b7ed4b1f1849be" Oct 30 17:45:04.658088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:04.658069 4622 scope.go:122] "RemoveContainer" containerID="6cb251f8c382cee2d6399e98b6959464db05230d6d7e0e39852dbeb9f5e63bd3" Oct 30 17:45:05.598138 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.598107 4622 scope.go:122] "RemoveContainer" containerID="137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441" Oct 30 17:45:05.598606 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.598522 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm_odh-ai-gateway-infra(acd26ed1-f9ee-4753-9838-06d7e1c15808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" Oct 30 17:45:05.602379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.602355 4622 scope.go:122] "RemoveContainer" containerID="925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb" Oct 30 17:45:05.602666 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.602640 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl_odh-ai-gateway-infra(a0b83092-530e-47fd-a57e-0c328da69269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" podUID="a0b83092-530e-47fd-a57e-0c328da69269" Oct 30 17:45:05.605468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.605435 4622 scope.go:122] "RemoveContainer" containerID="812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86" Oct 30 17:45:05.605797 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.605772 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn_odh-ai-gateway-infra(ba3befca-fc29-4461-a246-d1eff0234a7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" Oct 30 17:45:05.608043 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.608021 4622 scope.go:122] "RemoveContainer" containerID="977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23" Oct 30 17:45:05.608370 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.608342 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5_odh-ai-gateway-infra(2eb69423-836c-4e78-805b-fea52bbf1127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" Oct 30 17:45:05.610186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.610156 4622 scope.go:122] "RemoveContainer" containerID="5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896" Oct 30 17:45:05.610470 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.610442 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl_odh-ai-gateway-infra(d38439bc-b82d-4a24-95fd-9c9041c8b5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" Oct 30 17:45:05.611967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.611950 4622 scope.go:122] "RemoveContainer" containerID="f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8" Oct 30 17:45:05.612212 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.612193 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf_odh-ai-gateway-infra(b0f010a2-2a45-41f9-af45-9a9483de0032)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" Oct 30 17:45:05.613519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:05.613503 4622 scope.go:122] "RemoveContainer" containerID="c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058" Oct 30 17:45:05.613694 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:05.613678 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx_odh-ai-gateway-infra(2b137c19-15e4-4461-ac93-03cce05a72f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" Oct 30 17:45:14.367463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.367423 4622 scope.go:122] "RemoveContainer" containerID="c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058" Oct 30 17:45:14.368053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.367570 4622 scope.go:122] "RemoveContainer" containerID="137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441" Oct 30 17:45:14.368053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.367943 4622 scope.go:122] "RemoveContainer" containerID="812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86" Oct 30 17:45:14.368192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.368090 4622 scope.go:122] "RemoveContainer" containerID="925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb" Oct 30 17:45:14.388719 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.388695 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:45:14.388937 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:14.388890 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" podUID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" containerName="maas-api" containerID="cri-o://9d503a211b6b11141e65dbf027b7a4bb80281bc2c5aa0188f97cbb8e77a6d577" gracePeriod=30 Oct 30 17:45:15.023738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.023707 4622 generic.go:356] "Generic (PLEG): container finished" podID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" containerID="9d503a211b6b11141e65dbf027b7a4bb80281bc2c5aa0188f97cbb8e77a6d577" exitCode=0 Oct 30 17:45:15.023981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.023960 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" event={"ID":"fa9af145-6fd5-414b-88a8-eaeff045fa5c","Type":"ContainerDied","Data":"9d503a211b6b11141e65dbf027b7a4bb80281bc2c5aa0188f97cbb8e77a6d577"} Oct 30 17:45:15.056205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.055498 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:45:15.120338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.118787 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\") pod \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " Oct 30 17:45:15.120338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.118853 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\") pod \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " Oct 30 17:45:15.120338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.119031 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\") pod \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " Oct 30 17:45:15.120338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.119168 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\" (UniqueName: \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\") pod \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\" (UID: \"fa9af145-6fd5-414b-88a8-eaeff045fa5c\") " Oct 30 17:45:15.136253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.135127 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls" pod "fa9af145-6fd5-414b-88a8-eaeff045fa5c" (UID: "fa9af145-6fd5-414b-88a8-eaeff045fa5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:45:15.136253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.135483 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca" pod "fa9af145-6fd5-414b-88a8-eaeff045fa5c" (UID: "fa9af145-6fd5-414b-88a8-eaeff045fa5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:15.141497 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.141472 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs" pod "fa9af145-6fd5-414b-88a8-eaeff045fa5c" (UID: "fa9af145-6fd5-414b-88a8-eaeff045fa5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:45:15.141643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.141480 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9" pod "fa9af145-6fd5-414b-88a8-eaeff045fa5c" (UID: "fa9af145-6fd5-414b-88a8-eaeff045fa5c"). InnerVolumeSpecName "kube-api-access-5t8s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:15.223614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.223580 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8s9\" (UniqueName: \"kubernetes.io/projected/fa9af145-6fd5-414b-88a8-eaeff045fa5c-kube-api-access-5t8s9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:15.223614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.223611 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa9af145-6fd5-414b-88a8-eaeff045fa5c-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:15.223880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.223625 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:15.223880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.223637 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa9af145-6fd5-414b-88a8-eaeff045fa5c-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:15.366810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:15.366775 4622 scope.go:122] "RemoveContainer" containerID="5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896" Oct 30 17:45:16.035668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.035633 4622 generic.go:356] "Generic (PLEG): container finished" podID="a0b83092-530e-47fd-a57e-0c328da69269" containerID="25f3773e7f03170fb43a3e29af491d42b4d61eda16fec9da48c5acb9f565f641" exitCode=60 Oct 30 17:45:16.036107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.035704 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerDied","Data":"25f3773e7f03170fb43a3e29af491d42b4d61eda16fec9da48c5acb9f565f641"} Oct 30 17:45:16.036107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.035760 4622 scope.go:122] "RemoveContainer" containerID="925640808cb5f9a1f922aabc1afb8e8eaf499221cb39cff75096e8d95dc2b1eb" Oct 30 17:45:16.036233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.036165 4622 scope.go:122] "RemoveContainer" containerID="25f3773e7f03170fb43a3e29af491d42b4d61eda16fec9da48c5acb9f565f641" Oct 30 17:45:16.036531 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:16.036502 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl_odh-ai-gateway-infra(a0b83092-530e-47fd-a57e-0c328da69269)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" podUID="a0b83092-530e-47fd-a57e-0c328da69269" Oct 30 17:45:16.042405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.042383 4622 generic.go:356] "Generic (PLEG): container finished" podID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerID="c90f646e2139eef23804c68492ca3f23cb5112f9ad4a5d84f60a14cd2558304a" exitCode=60 Oct 30 17:45:16.042550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.042470 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerDied","Data":"c90f646e2139eef23804c68492ca3f23cb5112f9ad4a5d84f60a14cd2558304a"} Oct 30 17:45:16.042981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.042959 4622 scope.go:122] "RemoveContainer" containerID="c90f646e2139eef23804c68492ca3f23cb5112f9ad4a5d84f60a14cd2558304a" Oct 30 17:45:16.043360 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:16.043337 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn_odh-ai-gateway-infra(ba3befca-fc29-4461-a246-d1eff0234a7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" Oct 30 17:45:16.044635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.044618 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" Oct 30 17:45:16.044780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.044655 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw" event={"ID":"fa9af145-6fd5-414b-88a8-eaeff045fa5c","Type":"ContainerDied","Data":"ec002a8ac10ddbb373ff454fd80ddaff0441641d9b2bc311bf5923e27352b92a"} Oct 30 17:45:16.047356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.047326 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerStarted","Data":"b7258440ef747ea48039133b21768de5b9acf91740eb52da78653aa1acb5e913"} Oct 30 17:45:16.052322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.052294 4622 generic.go:356] "Generic (PLEG): container finished" podID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerID="4ab06630c95f8c303cf183981b66f6e1c0ecc92ebced5a65a8bfc5b5833a2d43" exitCode=60 Oct 30 17:45:16.052439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.052423 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerDied","Data":"4ab06630c95f8c303cf183981b66f6e1c0ecc92ebced5a65a8bfc5b5833a2d43"} Oct 30 17:45:16.052781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.052765 4622 scope.go:122] "RemoveContainer" containerID="4ab06630c95f8c303cf183981b66f6e1c0ecc92ebced5a65a8bfc5b5833a2d43" Oct 30 17:45:16.053045 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:16.053024 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx_odh-ai-gateway-infra(2b137c19-15e4-4461-ac93-03cce05a72f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" Oct 30 17:45:16.055648 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.055627 4622 generic.go:356] "Generic (PLEG): container finished" podID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerID="b26ffb7ea31496ad70a34b740a2960babf9c4dec39ae36434b7ae0dba467b224" exitCode=60 Oct 30 17:45:16.055734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.055692 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerDied","Data":"b26ffb7ea31496ad70a34b740a2960babf9c4dec39ae36434b7ae0dba467b224"} Oct 30 17:45:16.055947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.055930 4622 scope.go:122] "RemoveContainer" containerID="812f41ebbfc07b80523a2c50a1ad0d16f88911032ed77660a52b92c5696a3f86" Oct 30 17:45:16.056048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.055968 4622 scope.go:122] "RemoveContainer" containerID="b26ffb7ea31496ad70a34b740a2960babf9c4dec39ae36434b7ae0dba467b224" Oct 30 17:45:16.056236 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:16.056215 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm_odh-ai-gateway-infra(acd26ed1-f9ee-4753-9838-06d7e1c15808)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" Oct 30 17:45:16.070387 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.070366 4622 scope.go:122] "RemoveContainer" containerID="9d503a211b6b11141e65dbf027b7a4bb80281bc2c5aa0188f97cbb8e77a6d577" Oct 30 17:45:16.080485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.080460 4622 scope.go:122] "RemoveContainer" containerID="c3530e37b83a089de6c05941f3e19da69f7f760bde03b931b48d2a3230516058" Oct 30 17:45:16.093494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.093473 4622 scope.go:122] "RemoveContainer" containerID="137fd0a855bca5d541db11c05e782dc725a030104dbc9b6799fe60be03dd9441" Oct 30 17:45:16.192156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.192117 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:45:16.197556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.197531 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc4e96-77bfbdccbf-gfcqw"] Oct 30 17:45:16.376065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:16.376039 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" path="/var/lib/kubelet/pods/fa9af145-6fd5-414b-88a8-eaeff045fa5c/volumes" Oct 30 17:45:17.069537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.069505 4622 generic.go:356] "Generic (PLEG): container finished" podID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerID="b7258440ef747ea48039133b21768de5b9acf91740eb52da78653aa1acb5e913" exitCode=60 Oct 30 17:45:17.069971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.069579 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerDied","Data":"b7258440ef747ea48039133b21768de5b9acf91740eb52da78653aa1acb5e913"} Oct 30 17:45:17.069971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.069634 4622 scope.go:122] "RemoveContainer" containerID="5730202bafc68b0eebc66961185a5c4091669a466d4819527b218e9e05cae896" Oct 30 17:45:17.070080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.070045 4622 scope.go:122] "RemoveContainer" containerID="b7258440ef747ea48039133b21768de5b9acf91740eb52da78653aa1acb5e913" Oct 30 17:45:17.070410 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:17.070390 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl_odh-ai-gateway-infra(d38439bc-b82d-4a24-95fd-9c9041c8b5f8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" Oct 30 17:45:17.138409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.138377 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:17.184049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.184019 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:17.197052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.197018 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:17.269302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.269236 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:17.311706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.311685 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:17.339011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.338945 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:17.373996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.369424 4622 scope.go:122] "RemoveContainer" containerID="977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23" Oct 30 17:45:17.373996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.370984 4622 scope.go:122] "RemoveContainer" containerID="f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8" Oct 30 17:45:17.463961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.463930 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\" (UniqueName: \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\") pod \"a0b83092-530e-47fd-a57e-0c328da69269\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " Oct 30 17:45:17.464119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.463979 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\") pod \"a0b83092-530e-47fd-a57e-0c328da69269\" (UID: \"a0b83092-530e-47fd-a57e-0c328da69269\") " Oct 30 17:45:17.465925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.465892 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca" pod "a0b83092-530e-47fd-a57e-0c328da69269" (UID: "a0b83092-530e-47fd-a57e-0c328da69269"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:17.484523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.484482 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z" pod "a0b83092-530e-47fd-a57e-0c328da69269" (UID: "a0b83092-530e-47fd-a57e-0c328da69269"). InnerVolumeSpecName "kube-api-access-wf59z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:17.485927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.485906 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:17.488998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.488975 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:17.555149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.555125 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:17.566096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566073 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\" (UniqueName: \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\") pod \"2b137c19-15e4-4461-ac93-03cce05a72f1\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " Oct 30 17:45:17.566226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566113 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\") pod \"2b137c19-15e4-4461-ac93-03cce05a72f1\" (UID: \"2b137c19-15e4-4461-ac93-03cce05a72f1\") " Oct 30 17:45:17.566226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566171 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\") pod \"acd26ed1-f9ee-4753-9838-06d7e1c15808\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " Oct 30 17:45:17.566445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566421 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\" (UniqueName: \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\") pod \"acd26ed1-f9ee-4753-9838-06d7e1c15808\" (UID: \"acd26ed1-f9ee-4753-9838-06d7e1c15808\") " Oct 30 17:45:17.566573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566542 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca" pod "2b137c19-15e4-4461-ac93-03cce05a72f1" (UID: "2b137c19-15e4-4461-ac93-03cce05a72f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:17.566639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.566605 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca" pod "acd26ed1-f9ee-4753-9838-06d7e1c15808" (UID: "acd26ed1-f9ee-4753-9838-06d7e1c15808"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:17.567084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.567062 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd26ed1-f9ee-4753-9838-06d7e1c15808-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.567171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.567092 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf59z\" (UniqueName: \"kubernetes.io/projected/a0b83092-530e-47fd-a57e-0c328da69269-kube-api-access-wf59z\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.567171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.567108 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b83092-530e-47fd-a57e-0c328da69269-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.567171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.567122 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b137c19-15e4-4461-ac93-03cce05a72f1-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.569301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.568332 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6" pod "2b137c19-15e4-4461-ac93-03cce05a72f1" (UID: "2b137c19-15e4-4461-ac93-03cce05a72f1"). InnerVolumeSpecName "kube-api-access-dv7w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:17.569301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.568402 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5" pod "acd26ed1-f9ee-4753-9838-06d7e1c15808" (UID: "acd26ed1-f9ee-4753-9838-06d7e1c15808"). InnerVolumeSpecName "kube-api-access-ckgc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:17.668753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.668719 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\" (UniqueName: \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\") pod \"ba3befca-fc29-4461-a246-d1eff0234a7c\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " Oct 30 17:45:17.668947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.668795 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\") pod \"ba3befca-fc29-4461-a246-d1eff0234a7c\" (UID: \"ba3befca-fc29-4461-a246-d1eff0234a7c\") " Oct 30 17:45:17.669378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.669359 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv7w6\" (UniqueName: \"kubernetes.io/projected/2b137c19-15e4-4461-ac93-03cce05a72f1-kube-api-access-dv7w6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.669521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.669380 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgc5\" (UniqueName: \"kubernetes.io/projected/acd26ed1-f9ee-4753-9838-06d7e1c15808-kube-api-access-ckgc5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.671438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.669844 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca" pod "ba3befca-fc29-4461-a246-d1eff0234a7c" (UID: "ba3befca-fc29-4461-a246-d1eff0234a7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:17.672056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.672025 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6" pod "ba3befca-fc29-4461-a246-d1eff0234a7c" (UID: "ba3befca-fc29-4461-a246-d1eff0234a7c"). InnerVolumeSpecName "kube-api-access-mkwh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:17.770988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.770961 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwh6\" (UniqueName: \"kubernetes.io/projected/ba3befca-fc29-4461-a246-d1eff0234a7c-kube-api-access-mkwh6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:17.770988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:17.770988 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba3befca-fc29-4461-a246-d1eff0234a7c-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:18.122179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.122146 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerStarted","Data":"298a19f815b816237e70fba3a3cb30e82f10f81703aa55c93c4ff399c5ffe38a"} Oct 30 17:45:18.127058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.127027 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" event={"ID":"2b137c19-15e4-4461-ac93-03cce05a72f1","Type":"ContainerDied","Data":"8ef7967699e8f8b671dd9e74e8aef293785b07606e9bd7e1ad095cf96e91c531"} Oct 30 17:45:18.127317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.127299 4622 scope.go:122] "RemoveContainer" containerID="4ab06630c95f8c303cf183981b66f6e1c0ecc92ebced5a65a8bfc5b5833a2d43" Oct 30 17:45:18.127527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.127513 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx" Oct 30 17:45:18.134337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.133460 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" Oct 30 17:45:18.134337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.133537 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm" event={"ID":"acd26ed1-f9ee-4753-9838-06d7e1c15808","Type":"ContainerDied","Data":"a5b80c53236b9261fd5577754e3ea6d9f0165e189b7b2a4c30a29d6cccbcfb52"} Oct 30 17:45:18.135853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.135589 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" event={"ID":"a0b83092-530e-47fd-a57e-0c328da69269","Type":"ContainerDied","Data":"ede01fbf8332cf06aafcf57c4c16896f3df833ff1808684be927572786f551f6"} Oct 30 17:45:18.135853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.135610 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl" Oct 30 17:45:18.141309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.137297 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" Oct 30 17:45:18.141428 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.141405 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn" event={"ID":"ba3befca-fc29-4461-a246-d1eff0234a7c","Type":"ContainerDied","Data":"66da2ea38555e11b1a55cd7d62585660ec36725d6079df5c2b32ac569385c516"} Oct 30 17:45:18.146755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.145898 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerStarted","Data":"fe579ad3b7548de9e64586925f543603ffe356460a71201fb4851075ab7e3f2a"} Oct 30 17:45:18.152433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.151760 4622 scope.go:122] "RemoveContainer" containerID="b26ffb7ea31496ad70a34b740a2960babf9c4dec39ae36434b7ae0dba467b224" Oct 30 17:45:18.165936 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.165895 4622 scope.go:122] "RemoveContainer" containerID="25f3773e7f03170fb43a3e29af491d42b4d61eda16fec9da48c5acb9f565f641" Oct 30 17:45:18.178474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.178454 4622 scope.go:122] "RemoveContainer" containerID="c90f646e2139eef23804c68492ca3f23cb5112f9ad4a5d84f60a14cd2558304a" Oct 30 17:45:18.279051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.279018 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:18.291692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.291662 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-455086-29889705-qfwtl"] Oct 30 17:45:18.292439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.292416 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:18.322912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.322878 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:18.330546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.330514 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-143041-29889705-s84sx"] Oct 30 17:45:18.375956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.375883 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" path="/var/lib/kubelet/pods/2b137c19-15e4-4461-ac93-03cce05a72f1/volumes" Oct 30 17:45:18.376459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.376438 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b83092-530e-47fd-a57e-0c328da69269" path="/var/lib/kubelet/pods/a0b83092-530e-47fd-a57e-0c328da69269/volumes" Oct 30 17:45:18.405985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.405960 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:18.413240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.413214 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-506124-29889705-74btm"] Oct 30 17:45:18.436798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.436774 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\" (UniqueName: \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\") pod \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " Oct 30 17:45:18.437096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.437075 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\") pod \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\" (UID: \"d38439bc-b82d-4a24-95fd-9c9041c8b5f8\") " Oct 30 17:45:18.437473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.437448 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca" pod "d38439bc-b82d-4a24-95fd-9c9041c8b5f8" (UID: "d38439bc-b82d-4a24-95fd-9c9041c8b5f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:18.437614 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.437585 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:18.439343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.439319 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj" pod "d38439bc-b82d-4a24-95fd-9c9041c8b5f8" (UID: "d38439bc-b82d-4a24-95fd-9c9041c8b5f8"). InnerVolumeSpecName "kube-api-access-bsxdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:18.451464 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.451440 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:18.465007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.464982 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bba90b-29889705-fdgcn"] Oct 30 17:45:18.538901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:18.538868 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxdj\" (UniqueName: \"kubernetes.io/projected/d38439bc-b82d-4a24-95fd-9c9041c8b5f8-kube-api-access-bsxdj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:19.155147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.155114 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" Oct 30 17:45:19.155147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.155127 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl" event={"ID":"d38439bc-b82d-4a24-95fd-9c9041c8b5f8","Type":"ContainerDied","Data":"aa84d57c52cd5435336320fb1c59ec3f391c6c00364689940dde06f501c7228c"} Oct 30 17:45:19.155689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.155188 4622 scope.go:122] "RemoveContainer" containerID="b7258440ef747ea48039133b21768de5b9acf91740eb52da78653aa1acb5e913" Oct 30 17:45:19.157542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.157503 4622 generic.go:356] "Generic (PLEG): container finished" podID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerID="298a19f815b816237e70fba3a3cb30e82f10f81703aa55c93c4ff399c5ffe38a" exitCode=60 Oct 30 17:45:19.157658 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.157536 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerDied","Data":"298a19f815b816237e70fba3a3cb30e82f10f81703aa55c93c4ff399c5ffe38a"} Oct 30 17:45:19.158068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.158042 4622 scope.go:122] "RemoveContainer" containerID="298a19f815b816237e70fba3a3cb30e82f10f81703aa55c93c4ff399c5ffe38a" Oct 30 17:45:19.158444 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:19.158419 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf_odh-ai-gateway-infra(b0f010a2-2a45-41f9-af45-9a9483de0032)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" Oct 30 17:45:19.163911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.163889 4622 generic.go:356] "Generic (PLEG): container finished" podID="2eb69423-836c-4e78-805b-fea52bbf1127" containerID="fe579ad3b7548de9e64586925f543603ffe356460a71201fb4851075ab7e3f2a" exitCode=60 Oct 30 17:45:19.164009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.163931 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerDied","Data":"fe579ad3b7548de9e64586925f543603ffe356460a71201fb4851075ab7e3f2a"} Oct 30 17:45:19.164250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.164236 4622 scope.go:122] "RemoveContainer" containerID="fe579ad3b7548de9e64586925f543603ffe356460a71201fb4851075ab7e3f2a" Oct 30 17:45:19.164585 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:19.164551 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5_odh-ai-gateway-infra(2eb69423-836c-4e78-805b-fea52bbf1127)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" Oct 30 17:45:19.167390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.167369 4622 scope.go:122] "RemoveContainer" containerID="f2f65db60269e71614f4e8062db8fbf584ef1f68450a6633683ae8324fed4bb8" Oct 30 17:45:19.180744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.180729 4622 scope.go:122] "RemoveContainer" containerID="977e4da0c9653356021f37353e120400d08a4bc008b688fcd769efc615568a23" Oct 30 17:45:19.254288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.252788 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:19.308312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.307289 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-38f445-29889705-dbqwl"] Oct 30 17:45:19.323284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.311242 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:19.387425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:19.384658 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:20.492554 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.491542 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:20.504076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.504042 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\" (UniqueName: \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\") pod \"b0f010a2-2a45-41f9-af45-9a9483de0032\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " Oct 30 17:45:20.504076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.504092 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\") pod \"b0f010a2-2a45-41f9-af45-9a9483de0032\" (UID: \"b0f010a2-2a45-41f9-af45-9a9483de0032\") " Oct 30 17:45:20.505199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.504921 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca" pod "b0f010a2-2a45-41f9-af45-9a9483de0032" (UID: "b0f010a2-2a45-41f9-af45-9a9483de0032"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:20.517041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.516896 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg" pod "b0f010a2-2a45-41f9-af45-9a9483de0032" (UID: "b0f010a2-2a45-41f9-af45-9a9483de0032"). InnerVolumeSpecName "kube-api-access-q5fsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:20.536114 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.536081 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" path="/var/lib/kubelet/pods/acd26ed1-f9ee-4753-9838-06d7e1c15808/volumes" Oct 30 17:45:20.536994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.536972 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" path="/var/lib/kubelet/pods/ba3befca-fc29-4461-a246-d1eff0234a7c/volumes" Oct 30 17:45:20.537718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.537543 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" path="/var/lib/kubelet/pods/d38439bc-b82d-4a24-95fd-9c9041c8b5f8/volumes" Oct 30 17:45:20.563040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.563018 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:20.608924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.608888 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\" (UniqueName: \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\") pod \"2eb69423-836c-4e78-805b-fea52bbf1127\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " Oct 30 17:45:20.609088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.608953 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\") pod \"2eb69423-836c-4e78-805b-fea52bbf1127\" (UID: \"2eb69423-836c-4e78-805b-fea52bbf1127\") " Oct 30 17:45:20.609670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.609646 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5fsg\" (UniqueName: \"kubernetes.io/projected/b0f010a2-2a45-41f9-af45-9a9483de0032-kube-api-access-q5fsg\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:20.609670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.609669 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f010a2-2a45-41f9-af45-9a9483de0032-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:20.610316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.610290 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca" pod "2eb69423-836c-4e78-805b-fea52bbf1127" (UID: "2eb69423-836c-4e78-805b-fea52bbf1127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:45:20.626429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.621161 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s" pod "2eb69423-836c-4e78-805b-fea52bbf1127" (UID: "2eb69423-836c-4e78-805b-fea52bbf1127"). InnerVolumeSpecName "kube-api-access-dcb8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:45:20.710893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.710858 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcb8s\" (UniqueName: \"kubernetes.io/projected/2eb69423-836c-4e78-805b-fea52bbf1127-kube-api-access-dcb8s\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:20.710893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:20.710887 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb69423-836c-4e78-805b-fea52bbf1127-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:45:21.186181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.186140 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" event={"ID":"2eb69423-836c-4e78-805b-fea52bbf1127","Type":"ContainerDied","Data":"10a790c740f573f7f32b8f75392c22280dbe90e95b393e545b9e2ac34f46bf98"} Oct 30 17:45:21.186419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.186231 4622 scope.go:122] "RemoveContainer" containerID="fe579ad3b7548de9e64586925f543603ffe356460a71201fb4851075ab7e3f2a" Oct 30 17:45:21.186419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.186153 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5" Oct 30 17:45:21.189073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.189043 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" event={"ID":"b0f010a2-2a45-41f9-af45-9a9483de0032","Type":"ContainerDied","Data":"f9a13356a05db24c302633fea37efe6c17bbe5ab6551e166beb68b10629fb068"} Oct 30 17:45:21.189220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.189140 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf" Oct 30 17:45:21.205803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.205781 4622 scope.go:122] "RemoveContainer" containerID="298a19f815b816237e70fba3a3cb30e82f10f81703aa55c93c4ff399c5ffe38a" Oct 30 17:45:21.255456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.255330 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:21.263489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.262974 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-74762d-29889705-ccrg5"] Oct 30 17:45:21.307032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.306997 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:21.314282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:21.313659 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b1c345-29889705-p7wgf"] Oct 30 17:45:22.392384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:22.392342 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" path="/var/lib/kubelet/pods/2eb69423-836c-4e78-805b-fea52bbf1127/volumes" Oct 30 17:45:22.392868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:22.392837 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" path="/var/lib/kubelet/pods/b0f010a2-2a45-41f9-af45-9a9483de0032/volumes" Oct 30 17:45:24.486358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.486325 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:45:24.487047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487029 4622 cpu_manager.go:414] "containerMap: removing container" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487047 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487058 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487063 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487071 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487077 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487094 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487099 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487162 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487171 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487180 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487184 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487191 4622 cpu_manager.go:414] "containerMap: removing container" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487195 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487206 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487211 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487219 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487228 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487236 4622 cpu_manager.go:414] "containerMap: removing container" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487240 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487254 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487270 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487277 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487281 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487295 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487299 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487304 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487308 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487318 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487322 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487328 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487332 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487344 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487348 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487355 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487359 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487366 4622 cpu_manager.go:414] "containerMap: removing container" podUID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" containerName="maas-api" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487370 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" containerName="maas-api" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487478 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487485 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487492 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487498 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487505 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b83092-530e-47fd-a57e-0c328da69269" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487509 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487515 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9af145-6fd5-414b-88a8-eaeff045fa5c" containerName="maas-api" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487522 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487526 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487531 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487536 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487543 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b137c19-15e4-4461-ac93-03cce05a72f1" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487549 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487553 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487560 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487566 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487573 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487578 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:24.487606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.487583 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:24.492653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.492637 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.514892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.514862 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:45:24.529306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.520291 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-488c67-serving-cert\"" Oct 30 17:45:24.546610 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:24.541173 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:45:24.598383 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.598341 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.598550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.598394 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69bc\" (UniqueName: \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.598727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.598702 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.598797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.598754 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.713316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.703703 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.713316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.703769 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69bc\" (UniqueName: \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.713316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.704039 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.713316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.704086 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.713316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.704890 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.720028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.719237 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.721016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.720994 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.730087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.730055 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69bc\" (UniqueName: \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\") pod \"maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:24.806009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:24.805971 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:25.402311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:25.401202 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:45:25.404901 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:25.404856 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538963a2_cf66_4269_a2bc_30a8efb9b205.slice/crio-00637c1349d597a59ea485902d2efab3c4c840c2c38e6d5bb43bb7c472fa58f4 WatchSource:0}: Error finding container 00637c1349d597a59ea485902d2efab3c4c840c2c38e6d5bb43bb7c472fa58f4: Status 404 returned error can't find the container with id 00637c1349d597a59ea485902d2efab3c4c840c2c38e6d5bb43bb7c472fa58f4 Oct 30 17:45:26.288314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:26.286305 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" event={"ID":"538963a2-cf66-4269-a2bc-30a8efb9b205","Type":"ContainerStarted","Data":"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866"} Oct 30 17:45:26.288314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:26.286387 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" event={"ID":"538963a2-cf66-4269-a2bc-30a8efb9b205","Type":"ContainerStarted","Data":"00637c1349d597a59ea485902d2efab3c4c840c2c38e6d5bb43bb7c472fa58f4"} Oct 30 17:45:26.288314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:26.286433 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:26.335989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:26.335942 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" podStartSLOduration=1.870390305 podStartE2EDuration="2.335928984s" totalImagesPullingTime="465.538679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:26.331658039 +0000 UTC m=+1954.109129995" watchObservedRunningTime="2026-10-30 17:45:26.335928984 +0000 UTC m=+1954.113400939" Oct 30 17:45:29.032365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.032331 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs"] Oct 30 17:45:29.044016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.043989 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc4aa170-vmvjs"] Oct 30 17:45:29.775094 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775062 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:45:29.775766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775745 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:29.775892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775766 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:29.775892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775793 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:29.775892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775801 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb69423-836c-4e78-805b-fea52bbf1127" containerName="cleanup" Oct 30 17:45:29.775892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775846 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:29.775892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.775854 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f010a2-2a45-41f9-af45-9a9483de0032" containerName="cleanup" Oct 30 17:45:29.776140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.776032 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd26ed1-f9ee-4753-9838-06d7e1c15808" containerName="cleanup" Oct 30 17:45:29.776140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.776052 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38439bc-b82d-4a24-95fd-9c9041c8b5f8" containerName="cleanup" Oct 30 17:45:29.776140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.776064 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3befca-fc29-4461-a246-d1eff0234a7c" containerName="cleanup" Oct 30 17:45:29.780219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.780194 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.798051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.798020 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:45:29.805354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.804703 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1c345\"/\"openshift-service-ca.crt\"" Oct 30 17:45:29.805354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.804823 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1c345\"/\"default-dockercfg-b7mv9\"" Oct 30 17:45:29.805564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.805482 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1c345\"/\"kube-root-ca.crt\"" Oct 30 17:45:29.815308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.809726 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1c345\"/\"test-model-b1c345-kserve-self-signed-certs\"" Oct 30 17:45:29.891375 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891340 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.891529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891426 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.891586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891558 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.891633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891597 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.891730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891708 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.891954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.891923 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ndq\" (UniqueName: \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993421 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993380 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ndq\" (UniqueName: \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993528 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993602 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993740 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993723 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993752 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.993832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.993796 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.994237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.994046 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.994237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.994199 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.994451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.994343 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.996377 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.996354 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:29.996616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:29.996597 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:30.050314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:30.045077 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ndq\" (UniqueName: \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\") pod \"test-model-b1c345-kserve-78cf8cd967-gp8hx\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:30.104286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:30.103944 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:30.394458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:30.394386 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7466eeae-c194-42ea-ac6a-1fb3e8aa675a" path="/var/lib/kubelet/pods/7466eeae-c194-42ea-ac6a-1fb3e8aa675a/volumes" Oct 30 17:45:30.508746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:30.508717 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:45:30.509385 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:30.509359 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ed8eca0_ac15_44b3_9d77_2352d1204f3a.slice/crio-ff422c0eff2f741005f9c90204731987d07e390b51f0d1a1f5d3921735bdf450 WatchSource:0}: Error finding container ff422c0eff2f741005f9c90204731987d07e390b51f0d1a1f5d3921735bdf450: Status 404 returned error can't find the container with id ff422c0eff2f741005f9c90204731987d07e390b51f0d1a1f5d3921735bdf450 Oct 30 17:45:31.333153 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.333119 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" event={"ID":"3ed8eca0-ac15-44b3-9d77-2352d1204f3a","Type":"ContainerStarted","Data":"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace"} Oct 30 17:45:31.333711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.333164 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" event={"ID":"3ed8eca0-ac15-44b3-9d77-2352d1204f3a","Type":"ContainerStarted","Data":"ff422c0eff2f741005f9c90204731987d07e390b51f0d1a1f5d3921735bdf450"} Oct 30 17:45:31.334895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.334870 4622 generic.go:356] "Generic (PLEG): container finished" podID="1a116870-e9f7-41d5-98cc-828575727fd8" containerID="f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d" exitCode=0 Oct 30 17:45:31.335004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.334944 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerDied","Data":"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d"} Oct 30 17:45:31.335359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.335343 4622 scope.go:122] "RemoveContainer" containerID="f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d" Oct 30 17:45:31.336919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.336893 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:31.336981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.336959 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:31.337020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.336990 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:31.375277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.375230 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" podStartSLOduration=2.375217021 podStartE2EDuration="2.375217021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:31.374668358 +0000 UTC m=+1959.152140335" watchObservedRunningTime="2026-10-30 17:45:31.375217021 +0000 UTC m=+1959.152688976" Oct 30 17:45:31.727106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:31.727021 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:45:32.104879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.104849 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:32.312803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.312774 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:45:32.353590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.353481 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerStarted","Data":"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96"} Oct 30 17:45:32.521161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.521083 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:45:32.522579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.522556 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:32.522710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:32.522599 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:33.521917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:33.521881 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:33.522334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:33.521936 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:34.382730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.382694 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:45:34.388145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.388112 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.400198 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400160 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.400365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400256 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.400365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400306 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9wp\" (UniqueName: \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.400737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400712 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.400884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400862 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.401028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.400980 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.404686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.404663 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:45:34.411461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.411425 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-874b83\"/\"openshift-service-ca.crt\"" Oct 30 17:45:34.411461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.411426 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-874b83\"/\"kube-root-ca.crt\"" Oct 30 17:45:34.414998 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.414978 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-874b83\"/\"default-dockercfg-rlvj9\"" Oct 30 17:45:34.415181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.415158 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-874b83\"/\"test-model-874b83-kserve-self-signed-certs\"" Oct 30 17:45:34.502968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.502928 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503018 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503055 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503155 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503229 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503274 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9wp\" (UniqueName: \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.503964 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.503938 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.504171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.504148 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.504306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.504199 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.505992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.505967 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.508506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.508483 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.521665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.521637 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:34.521790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.521695 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:34.527031 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.527008 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9wp\" (UniqueName: \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\") pod \"test-model-874b83-kserve-5874fbcb8c-pm2vh\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:34.715338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:34.711498 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:35.172304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.172196 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:45:35.174794 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:35.174765 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf264a5d_c7ea_4fc7_855f_93fc980c8da1.slice/crio-f563c181cb9810cce8f871b1a9aeeb35d1498e8fe5314d58e3994893fb9f0540 WatchSource:0}: Error finding container f563c181cb9810cce8f871b1a9aeeb35d1498e8fe5314d58e3994893fb9f0540: Status 404 returned error can't find the container with id f563c181cb9810cce8f871b1a9aeeb35d1498e8fe5314d58e3994893fb9f0540 Oct 30 17:45:35.392091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.392052 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" event={"ID":"bf264a5d-c7ea-4fc7-855f-93fc980c8da1","Type":"ContainerStarted","Data":"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d"} Oct 30 17:45:35.392275 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.392112 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" event={"ID":"bf264a5d-c7ea-4fc7-855f-93fc980c8da1","Type":"ContainerStarted","Data":"f563c181cb9810cce8f871b1a9aeeb35d1498e8fe5314d58e3994893fb9f0540"} Oct 30 17:45:35.435159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.435057 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" podStartSLOduration=1.435044844 podStartE2EDuration="1.435044844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:35.432209872 +0000 UTC m=+1963.209681828" watchObservedRunningTime="2026-10-30 17:45:35.435044844 +0000 UTC m=+1963.212516822" Oct 30 17:45:35.521376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.521343 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:35.521538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.521393 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:35.712234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:35.712157 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:36.521414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:36.521378 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:36.521688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:36.521660 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:37.521292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:37.521239 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 17:45:37.521821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:37.521313 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 17:45:38.527972 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:38.527940 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:45:38.528506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:38.528383 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:45:38.529133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:38.529107 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:45:39.857434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:39.857400 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:45:39.861952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:39.861934 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:39.881274 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:39.881242 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-143041-openshift-default-dockercfg-j9lzf\"" Oct 30 17:45:39.897299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:39.897235 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:45:40.029761 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.029728 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.029919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.029788 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.029967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.029916 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.029964 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.030002 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.030088 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.030112 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.030191 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6snp\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.030246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.030222 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.105702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.105672 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:40.117911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.117859 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:40.131717 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131695 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.131813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131758 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.131875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131810 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.131875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131864 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.131979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131902 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.131993 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132029 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132075 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132108 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6snp\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132151 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132345 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132491 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132567 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.132771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.132750 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.134231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.134212 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.134401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.134384 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.158799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.158776 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.163483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.163464 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6snp\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\") pod \"e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.177157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.177115 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:40.453276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.453189 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:45:40.497086 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:40.497059 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fffe92d_deb6_4993_8111_88e79bab77a7.slice/crio-e981f06478b4c3266d4abaff438ebcdc8bb35c07d463efb83601dd5b8f47944e WatchSource:0}: Error finding container e981f06478b4c3266d4abaff438ebcdc8bb35c07d463efb83601dd5b8f47944e: Status 404 returned error can't find the container with id e981f06478b4c3266d4abaff438ebcdc8bb35c07d463efb83601dd5b8f47944e Oct 30 17:45:40.497215 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.497135 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:45:40.500098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.500069 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:40.500178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.500136 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:40.500178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:40.500164 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:41.450500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:41.450456 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" event={"ID":"4fffe92d-deb6-4993-8111-88e79bab77a7","Type":"ContainerStarted","Data":"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5"} Oct 30 17:45:41.451009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:41.450544 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" event={"ID":"4fffe92d-deb6-4993-8111-88e79bab77a7","Type":"ContainerStarted","Data":"e981f06478b4c3266d4abaff438ebcdc8bb35c07d463efb83601dd5b8f47944e"} Oct 30 17:45:41.496984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:41.496934 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podStartSLOduration=2.496920696 podStartE2EDuration="2.496920696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:41.494420685 +0000 UTC m=+1969.271892691" watchObservedRunningTime="2026-10-30 17:45:41.496920696 +0000 UTC m=+1969.274392651" Oct 30 17:45:42.178305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:42.178247 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:42.179782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:42.179738 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:42.179913 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:42.179812 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:43.177850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.177814 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:43.178246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.177868 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:43.278255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.278219 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:45:43.283118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.283096 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.296520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.296490 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.296737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.296711 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.296851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.296817 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5fg\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.296929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.296907 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.296987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.296971 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.297027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.297017 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.297079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.297041 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.297218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.297195 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.297305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.297252 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.309938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.309915 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:45:43.316837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.316820 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bba90b-openshift-default-dockercfg-d9vst\"" Oct 30 17:45:43.398411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398380 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398418 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398470 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5fg\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398509 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398542 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398576 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398599 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398669 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.398997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.398707 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.399057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.399001 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.399100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.399074 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.399136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.399108 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.399169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.399149 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.399420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.399393 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.401032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.401004 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.401123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.401112 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.423519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.423498 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5fg\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.423617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.423558 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\") pod \"e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.598954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.598925 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:43.967885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.967842 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:45:43.969488 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:43.969461 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fe6e46_5076_4fe0_996a_a8d9be8359a4.slice/crio-363c61f39901df89a3c306adce9857cf904571d720654ca3f333d3d35f34a28f WatchSource:0}: Error finding container 363c61f39901df89a3c306adce9857cf904571d720654ca3f333d3d35f34a28f: Status 404 returned error can't find the container with id 363c61f39901df89a3c306adce9857cf904571d720654ca3f333d3d35f34a28f Oct 30 17:45:43.972537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.972499 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:43.972623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.972570 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:43.972623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:43.972600 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:45:44.177982 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.177939 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:44.178519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.177999 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:44.492788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.492759 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" event={"ID":"53fe6e46-5076-4fe0-996a-a8d9be8359a4","Type":"ContainerStarted","Data":"095cf7fcbed4e21ba37cc73f0d5593aa3974d0fb5f03cc3a1e13fd74986c3ee9"} Oct 30 17:45:44.492955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.492807 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" event={"ID":"53fe6e46-5076-4fe0-996a-a8d9be8359a4","Type":"ContainerStarted","Data":"363c61f39901df89a3c306adce9857cf904571d720654ca3f333d3d35f34a28f"} Oct 30 17:45:44.579867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.579791 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podStartSLOduration=1.5797648020000001 podStartE2EDuration="1.579764802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:44.539698314 +0000 UTC m=+1972.317170303" watchObservedRunningTime="2026-10-30 17:45:44.579764802 +0000 UTC m=+1972.357236757" Oct 30 17:45:44.599166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.599135 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:44.712818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.712776 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:44.775343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:44.770550 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:45.190446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:45.190406 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:45.191207 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:45.191173 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:45.518636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:45.518565 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:45:45.606082 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:45.606041 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:45.606405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:45.606373 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:46.181606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:46.181562 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:46.181815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:46.181638 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:46.599509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:46.599463 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:46.600001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:46.599532 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:47.177772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:47.177732 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 17:45:47.177967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:47.177789 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 17:45:47.599848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:47.599808 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:47.600295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:47.599874 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:48.182596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:48.182565 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:48.182972 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:48.182949 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:48.183805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:48.183788 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:45:48.599815 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:48.599782 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:48.599986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:48.599834 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:49.601688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:49.601651 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:49.602489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:49.602409 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:50.599783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:50.599748 4622 patch_prober.go:29] interesting pod/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 17:45:50.599958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:50.599799 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 17:45:51.717954 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:51.717912 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:51.718504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:51.718287 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:51.720084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:51.720060 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:45:54.771288 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:45:54.771232 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:45:55.678609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.678569 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:45:55.685892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.685868 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704112 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zv9\" (UniqueName: \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704223 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704342 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704370 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704503 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.704808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.704540 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.707125 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.707097 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:45:55.715289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.711834 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee9ff3\"/\"default-dockercfg-lsmzh\"" Oct 30 17:45:55.715289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.714004 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee9ff3\"/\"test-model-ee9ff3-kserve-self-signed-certs\"" Oct 30 17:45:55.715289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.714406 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee9ff3\"/\"openshift-service-ca.crt\"" Oct 30 17:45:55.715289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.715069 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee9ff3\"/\"kube-root-ca.crt\"" Oct 30 17:45:55.806291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.805871 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.806774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.806738 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.806944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.806925 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.806972 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.807062 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zv9\" (UniqueName: \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.807114 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.807147 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.807406 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.807689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.807670 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.809407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.809385 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.810067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.810048 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:55.831178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:55.831150 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zv9\" (UniqueName: \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\") pod \"test-model-ee9ff3-kserve-6f7f558b55-2x27q\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:56.049282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:56.038590 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:56.473497 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:56.473469 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda0037bb_2076_4137_aa71_e7d7e0e4bf3d.slice/crio-417d983b7aed6b0650da79474f553f09722f7db8c2c588a5bfef0abe89868c63 WatchSource:0}: Error finding container 417d983b7aed6b0650da79474f553f09722f7db8c2c588a5bfef0abe89868c63: Status 404 returned error can't find the container with id 417d983b7aed6b0650da79474f553f09722f7db8c2c588a5bfef0abe89868c63 Oct 30 17:45:56.477435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:56.476824 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:45:56.733487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:56.733399 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" event={"ID":"da0037bb-2076-4137-aa71-e7d7e0e4bf3d","Type":"ContainerStarted","Data":"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253"} Oct 30 17:45:56.733708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:56.733687 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" event={"ID":"da0037bb-2076-4137-aa71-e7d7e0e4bf3d","Type":"ContainerStarted","Data":"417d983b7aed6b0650da79474f553f09722f7db8c2c588a5bfef0abe89868c63"} Oct 30 17:45:56.788161 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:56.788099 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" podStartSLOduration=1.788085787 podStartE2EDuration="1.788085787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:56.784980235 +0000 UTC m=+1984.562452191" watchObservedRunningTime="2026-10-30 17:45:56.788085787 +0000 UTC m=+1984.565557742" Oct 30 17:45:57.039603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:57.039581 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:45:58.687173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.687134 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:45:58.701764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.701727 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.710801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.710775 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:45:58.728932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.728907 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f8b48d\"/\"default-dockercfg-cjgl9\"" Oct 30 17:45:58.729104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.728936 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f8b48d\"/\"test-model-f8b48d-kserve-self-signed-certs\"" Oct 30 17:45:58.729181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.728952 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f8b48d\"/\"kube-root-ca.crt\"" Oct 30 17:45:58.729181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.729032 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f8b48d\"/\"openshift-service-ca.crt\"" Oct 30 17:45:58.763767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.763683 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.763767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.763728 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.763767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.763751 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8tq\" (UniqueName: \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.764093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.763939 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.764093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.763976 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.764093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.764084 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865527 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865568 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8tq\" (UniqueName: \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865676 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865712 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865829 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.865923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.865898 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.866472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.866447 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.866620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.866550 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.866620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.866602 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.868552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.868519 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.868736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.868717 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:58.888598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:58.888573 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8tq\" (UniqueName: \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\") pod \"test-model-f8b48d-kserve-54dddcb5f5-fj6dc\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:59.027358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:59.027221 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:45:59.386034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:59.386000 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:45:59.386931 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:45:59.386896 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd036cba4_d86e_4c12_84cd_541e81e99455.slice/crio-0329c497cf99bc354e38d9f82d1111ee2509e2050c889b8d9e1e862c2fae29bd WatchSource:0}: Error finding container 0329c497cf99bc354e38d9f82d1111ee2509e2050c889b8d9e1e862c2fae29bd: Status 404 returned error can't find the container with id 0329c497cf99bc354e38d9f82d1111ee2509e2050c889b8d9e1e862c2fae29bd Oct 30 17:45:59.762594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:59.762502 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" event={"ID":"d036cba4-d86e-4c12-84cd-541e81e99455","Type":"ContainerStarted","Data":"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477"} Oct 30 17:45:59.762594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:59.762552 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" event={"ID":"d036cba4-d86e-4c12-84cd-541e81e99455","Type":"ContainerStarted","Data":"0329c497cf99bc354e38d9f82d1111ee2509e2050c889b8d9e1e862c2fae29bd"} Oct 30 17:45:59.807636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:45:59.807579 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" podStartSLOduration=1.807564248 podStartE2EDuration="1.807564248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:45:59.805501103 +0000 UTC m=+1987.582973060" watchObservedRunningTime="2026-10-30 17:45:59.807564248 +0000 UTC m=+1987.585036202" Oct 30 17:46:00.028317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.028201 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:46:00.675987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.675951 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:46:00.681394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.681368 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.710305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.704082 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:46:00.710305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.709457 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24fb0d\"/\"default-dockercfg-2wjsr\"" Oct 30 17:46:00.710305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.709617 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24fb0d\"/\"openshift-service-ca.crt\"" Oct 30 17:46:00.710305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.709714 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24fb0d\"/\"kube-root-ca.crt\"" Oct 30 17:46:00.710305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.709840 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24fb0d\"/\"test-model-24fb0d-kserve-self-signed-certs\"" Oct 30 17:46:00.815645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.815597 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.816139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.815742 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.816139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.815823 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtt48\" (UniqueName: \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.816139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.815898 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.816139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.815949 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.816139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.816068 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917318 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917418 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917523 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917588 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917686 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.917817 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtt48\" (UniqueName: \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.921578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.918598 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.932078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.926153 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.933653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.933621 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.947671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.947638 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.948521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.948494 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:00.995755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:00.994708 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtt48\" (UniqueName: \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\") pod \"test-model-24fb0d-kserve-9cc68795f-4v7kf\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:01.003720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:01.003430 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:01.498399 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:46:01.498357 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29177b02_a141_4690_8f91_7978b2721f45.slice/crio-8b0b0af593adb10a798b0619be15cd696c1a39585583caef77f39249536a5d5d WatchSource:0}: Error finding container 8b0b0af593adb10a798b0619be15cd696c1a39585583caef77f39249536a5d5d: Status 404 returned error can't find the container with id 8b0b0af593adb10a798b0619be15cd696c1a39585583caef77f39249536a5d5d Oct 30 17:46:01.498928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:01.498903 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:46:01.789784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:01.789748 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" event={"ID":"29177b02-a141-4690-8f91-7978b2721f45","Type":"ContainerStarted","Data":"d403f3e0631fd29c177e44973286dda599115f61cef8b58ac894c4642e1a3d9b"} Oct 30 17:46:01.789978 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:01.789796 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" event={"ID":"29177b02-a141-4690-8f91-7978b2721f45","Type":"ContainerStarted","Data":"8b0b0af593adb10a798b0619be15cd696c1a39585583caef77f39249536a5d5d"} Oct 30 17:46:01.834727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:01.834672 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" podStartSLOduration=1.834653248 podStartE2EDuration="1.834653248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:01.830447005 +0000 UTC m=+1989.607918962" watchObservedRunningTime="2026-10-30 17:46:01.834653248 +0000 UTC m=+1989.612125209" Oct 30 17:46:02.004668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:02.004624 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:02.337457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:02.337210 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:46:02.339504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:02.337592 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" containerID="cri-o://f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb" gracePeriod=30 Oct 30 17:46:03.879283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.878899 4622 generic.go:356] "Generic (PLEG): container finished" podID="d8df3750-cc02-42da-8e32-694298a39b89" containerID="5694eb7dbd5af0866d7d2134c3892e7c2ff006932573b59a31853a8a7f532dd0" exitCode=0 Oct 30 17:46:03.883281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.880086 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerDied","Data":"5694eb7dbd5af0866d7d2134c3892e7c2ff006932573b59a31853a8a7f532dd0"} Oct 30 17:46:03.883281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.880523 4622 scope.go:122] "RemoveContainer" containerID="5694eb7dbd5af0866d7d2134c3892e7c2ff006932573b59a31853a8a7f532dd0" Oct 30 17:46:03.928294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.894786 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:46:03.928294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.894899 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:46:03.928294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:03.894943 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:46:04.890921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:04.890881 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerStarted","Data":"f5b322cb1cc27c0cf962d79cc5c8394a06ba00d06fd64e03b0469ca4a2684b2b"} Oct 30 17:46:05.128306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:05.128244 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:46:05.129706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:05.129678 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:05.129855 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:05.129731 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:06.039330 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:06.039296 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:46:06.079677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:06.079650 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:46:06.130097 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:06.130063 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:06.130804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:06.130395 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:06.920710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:06.920682 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:46:07.128159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.128124 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:07.128577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.128179 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:07.678493 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.678468 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:46:07.720225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720189 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720295 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720377 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720411 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720458 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.720646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720539 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720574 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720598 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.720646 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720638 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720666 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.720912 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.720769 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\") pod \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\" (UID: \"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d\") " Oct 30 17:46:07.721627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.721358 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.721627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.721413 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.721627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.721432 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.721627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.721566 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.721627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.721590 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:46:07.727586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.723970 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:46:07.730710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.730664 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:46:07.732384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.732307 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "kube-api-access-lnsqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.732586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.732560 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token" pod "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" (UID: "b111a1d6-5c4a-4c3a-b6ff-62313e762b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:46:07.823224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823185 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823221 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823235 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnsqx\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-kube-api-access-lnsqx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823251 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823278 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823293 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.823517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.823306 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:46:07.918110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.918026 4622 generic.go:356] "Generic (PLEG): container finished" podID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerID="f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb" exitCode=0 Oct 30 17:46:07.918110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.918105 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" Oct 30 17:46:07.918380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.918112 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" event={"ID":"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d","Type":"ContainerDied","Data":"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb"} Oct 30 17:46:07.918380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.918188 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf" event={"ID":"b111a1d6-5c4a-4c3a-b6ff-62313e762b6d","Type":"ContainerDied","Data":"0101fcd8409565790c669a3a552c16426dafe2b34136ea496c7ccc73fa578850"} Oct 30 17:46:07.918380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.918211 4622 scope.go:122] "RemoveContainer" containerID="f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb" Oct 30 17:46:07.930695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.930673 4622 scope.go:122] "RemoveContainer" containerID="f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb" Oct 30 17:46:07.930961 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:46:07.930941 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb\": container with ID starting with f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb not found: ID does not exist" containerID="f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb" Oct 30 17:46:07.931016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.930972 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb"} err="failed to get container status \"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb\": rpc error: code = NotFound desc = could not find container \"f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb\": container with ID starting with f958050e18536fbd99b79f3f762a8cee1d1d3a3eac29ca9b32a8e85e7cc34deb not found: ID does not exist" Oct 30 17:46:07.981623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.981585 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:46:07.985622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:07.985584 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-74762d-openshift-default-5f5f9cdd99-jxrtf"] Oct 30 17:46:08.131287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:08.129666 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:08.131287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:08.129725 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:08.314687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:08.314649 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:46:08.377527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:08.377491 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" path="/var/lib/kubelet/pods/b111a1d6-5c4a-4c3a-b6ff-62313e762b6d/volumes" Oct 30 17:46:09.027533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:09.027499 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:46:09.041049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:09.041022 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:46:09.127524 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:09.127491 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:09.127720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:09.127558 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:10.189284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:10.179536 4622 patch_prober.go:29] interesting pod/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 17:46:10.189284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:10.179618 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 17:46:10.229461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:10.229429 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:46:11.004741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:11.004707 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:11.017827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:11.017801 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:11.182759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:11.182729 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:46:11.240789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:11.240753 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:46:11.242007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:11.241985 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:46:24.534606 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:46:24.534531 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:46:54.242140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:54.242111 4622 scope.go:122] "RemoveContainer" containerID="a6bd2178a173370ee29b7b6d4c06226d222df77d9ad6760881bb3117a442ac4f" Oct 30 17:46:54.252478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:46:54.252460 4622 scope.go:122] "RemoveContainer" containerID="039972031a92faba3621d3699e668c2bc6059ee6aeddc3948cae13fe7188b2c4" Oct 30 17:46:54.535320 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:46:54.535251 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:47:24.534795 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:47:24.534758 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:47:36.671155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.671113 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:47:36.672091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.672070 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" Oct 30 17:47:36.672091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.672092 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" Oct 30 17:47:36.672247 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.672237 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b111a1d6-5c4a-4c3a-b6ff-62313e762b6d" containerName="istio-proxy" Oct 30 17:47:36.675850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.675827 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.713824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.713791 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:47:36.722845 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.722806 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8fc\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723013 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.722900 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723013 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.722936 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723097 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723033 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723128 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723303 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723245 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723342 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723374 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.723436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.723404 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.777467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.777425 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-043253-openshift-default-dockercfg-vlpqj\"" Oct 30 17:47:36.825143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825108 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825191 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825315 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825344 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825372 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825392 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825491 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8fc\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825557 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825586 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825591 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825821 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825865 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.825960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.825897 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.826026 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.826008 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.828029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.828001 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.828135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.828116 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.848666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.848635 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.848816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.848736 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8fc\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-pj2jk\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:36.995700 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:36.995620 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:37.317674 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:47:37.317638 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9392dd6_86d2_498b_8a76_c84e06392bf8.slice/crio-21ac3f023043e8e823da7c03608b5f9a60c6ca7c7ebe564cb71944303734cf21 WatchSource:0}: Error finding container 21ac3f023043e8e823da7c03608b5f9a60c6ca7c7ebe564cb71944303734cf21: Status 404 returned error can't find the container with id 21ac3f023043e8e823da7c03608b5f9a60c6ca7c7ebe564cb71944303734cf21 Oct 30 17:47:37.318029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.318005 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:47:37.320602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.320568 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:37.320692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.320639 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:37.320692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.320668 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:37.950152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.950118 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" event={"ID":"c9392dd6-86d2-498b-8a76-c84e06392bf8","Type":"ContainerStarted","Data":"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656"} Oct 30 17:47:37.950630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.950171 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" event={"ID":"c9392dd6-86d2-498b-8a76-c84e06392bf8","Type":"ContainerStarted","Data":"21ac3f023043e8e823da7c03608b5f9a60c6ca7c7ebe564cb71944303734cf21"} Oct 30 17:47:37.996040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:37.996012 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:38.021681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:38.021629 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podStartSLOduration=2.021614712 podStartE2EDuration="2.021614712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:38.008353196 +0000 UTC m=+2085.785825147" watchObservedRunningTime="2026-10-30 17:47:38.021614712 +0000 UTC m=+2085.799086666" Oct 30 17:47:38.997724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:38.997694 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:38.998131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:38.997744 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:39.996909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:39.996874 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:39.997121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:39.996934 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:40.996450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:40.996411 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:40.996831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:40.996495 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:41.996227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:41.996191 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:41.996435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:41.996276 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:42.996163 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:42.996130 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:42.996565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:42.996182 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:43.478309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.478274 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:47:43.483129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.483109 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.505295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.505247 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:47:43.516230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.516202 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-303ba5-openshift-default-dockercfg-d5wdq\"" Oct 30 17:47:43.576980 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.576945 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtt4\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.576986 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577069 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577175 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577245 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577350 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577382 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577405 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.577551 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.577536 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.678821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.678793 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.678994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.678842 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.678994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.678873 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.678994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.678895 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679004 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679155 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtt4\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679186 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679228 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679329 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679360 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679400 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679489 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679725 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679704 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.679797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.679775 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.681468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.681447 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.681808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.681785 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.701635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.701614 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtt4\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.701857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.701839 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\") pod \"e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.821063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.821031 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:43.996245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.996211 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:47:43.996883 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:43.996848 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 17:47:44.172398 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:47:44.172360 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cc1f6d_f714_4171_ab66_9d97a342da42.slice/crio-fff6bda0efafde2d8c856b2cf987354642c18e9273c848c7e1ae74a58639ef25 WatchSource:0}: Error finding container fff6bda0efafde2d8c856b2cf987354642c18e9273c848c7e1ae74a58639ef25: Status 404 returned error can't find the container with id fff6bda0efafde2d8c856b2cf987354642c18e9273c848c7e1ae74a58639ef25 Oct 30 17:47:44.174572 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:44.174543 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:47:44.174753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:44.174722 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:44.174797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:44.174785 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:44.174840 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:44.174814 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:45.001602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.001573 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:45.002107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.002087 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:45.002653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.002633 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:47:45.024561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.024528 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" event={"ID":"d1cc1f6d-f714-4171-ab66-9d97a342da42","Type":"ContainerStarted","Data":"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030"} Oct 30 17:47:45.024713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.024574 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" event={"ID":"d1cc1f6d-f714-4171-ab66-9d97a342da42","Type":"ContainerStarted","Data":"fff6bda0efafde2d8c856b2cf987354642c18e9273c848c7e1ae74a58639ef25"} Oct 30 17:47:45.143536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.143469 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podStartSLOduration=2.143449876 podStartE2EDuration="2.143449876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:45.137971586 +0000 UTC m=+2092.915443542" watchObservedRunningTime="2026-10-30 17:47:45.143449876 +0000 UTC m=+2092.920921831" Oct 30 17:47:45.821830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.821798 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:45.823142 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.823115 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:45.823285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:45.823168 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:46.821345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:46.821311 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:46.821711 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:46.821363 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:47.821853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:47.821798 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:47.822506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:47.821903 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:48.822188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:48.822140 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:48.822632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:48.822216 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:49.822384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:49.822350 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:49.822787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:49.822409 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:50.821362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:50.821332 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 17:47:50.821544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:50.821392 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 17:47:51.666818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.666783 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:47:51.671444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.671420 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.700352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.700320 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:47:51.789120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789087 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789148 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789236 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789359 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789405 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpc8\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789478 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789615 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789598 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789642 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.789753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.789738 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.826557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.826526 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:51.826911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.826891 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:51.827623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.827603 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:47:51.891048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891013 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891073 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891110 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891221 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891202 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891253 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpc8\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891456 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891362 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891457 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891512 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891579 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.891898 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891869 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.892002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.891979 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.892066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.892037 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.892139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.892058 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.892405 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.892379 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.894204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.894182 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.894999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.894976 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.940030 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.939969 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.940185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.940162 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpc8\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-kz4qj\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:51.990046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:51.989762 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:52.508234 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:47:52.508205 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc84ae49_4e6e_4360_8c52_eb38ab7f3ab3.slice/crio-611e43b74c2a2f052bca9d8cb0576359c57df6beb52f1f70c04617ee8e484aa3 WatchSource:0}: Error finding container 611e43b74c2a2f052bca9d8cb0576359c57df6beb52f1f70c04617ee8e484aa3: Status 404 returned error can't find the container with id 611e43b74c2a2f052bca9d8cb0576359c57df6beb52f1f70c04617ee8e484aa3 Oct 30 17:47:52.508721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:52.508700 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:47:52.511206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:52.511168 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:52.511353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:52.511248 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:52.511353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:52.511318 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:47:53.170072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.164735 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" event={"ID":"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3","Type":"ContainerStarted","Data":"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7"} Oct 30 17:47:53.170072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.164796 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" event={"ID":"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3","Type":"ContainerStarted","Data":"611e43b74c2a2f052bca9d8cb0576359c57df6beb52f1f70c04617ee8e484aa3"} Oct 30 17:47:53.232200 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.232143 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podStartSLOduration=2.2321234309999998 podStartE2EDuration="2.232123431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:53.227056609 +0000 UTC m=+2101.004528565" watchObservedRunningTime="2026-10-30 17:47:53.232123431 +0000 UTC m=+2101.009595386" Oct 30 17:47:53.993436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.991082 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:54.000693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.998636 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:54.000693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:53.998705 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:54.579934 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:47:54.579898 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:47:54.991331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:54.990290 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:54.991331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:54.990356 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:55.990970 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:55.990935 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:55.991485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:55.990988 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:56.991003 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:56.990968 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:56.991509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:56.991026 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:57.991096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:57.991056 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:57.991548 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:57.991124 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:58.990332 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:58.990296 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 17:47:58.990547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:58.990348 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 17:47:59.995090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:59.995060 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:59.995541 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:59.995523 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:47:59.996205 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:47:59.996186 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:48:06.671430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.671392 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:48:06.676000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.675977 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.724710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.724668 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:48:06.765039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765008 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765044 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765084 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765418 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765538 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765580 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765642 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcg9\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765700 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.765790 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.765768 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.866900 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.866870 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.866937 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.866968 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.866987 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcg9\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867131 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867166 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867225 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867276 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867258 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867353 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867366 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867500 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867599 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.867897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.867877 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.869591 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.869568 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.870988 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.870970 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.897980 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.897954 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.898341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.898325 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcg9\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\") pod \"e2e-inf-a-043253-openshift-default-58965447bf-5ttkv\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:06.992457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:06.992406 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:07.367369 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:48:07.367341 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52fa5f6_44d5_4412_b1c5_c98fa850c47b.slice/crio-cb30be12ae5252963cc843a760bcfbe26923141b2c9bf32f9852264463cbc6a5 WatchSource:0}: Error finding container cb30be12ae5252963cc843a760bcfbe26923141b2c9bf32f9852264463cbc6a5: Status 404 returned error can't find the container with id cb30be12ae5252963cc843a760bcfbe26923141b2c9bf32f9852264463cbc6a5 Oct 30 17:48:07.368649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:07.368624 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:48:07.370318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:07.370283 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:48:07.370410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:07.370349 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:48:07.370410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:07.370377 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:48:08.380483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.380447 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" event={"ID":"e52fa5f6-44d5-4412-b1c5-c98fa850c47b","Type":"ContainerStarted","Data":"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b"} Oct 30 17:48:08.380881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.380505 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" event={"ID":"e52fa5f6-44d5-4412-b1c5-c98fa850c47b","Type":"ContainerStarted","Data":"cb30be12ae5252963cc843a760bcfbe26923141b2c9bf32f9852264463cbc6a5"} Oct 30 17:48:08.432942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.432896 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podStartSLOduration=2.432881436 podStartE2EDuration="2.432881436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:08.430059252 +0000 UTC m=+2116.207531207" watchObservedRunningTime="2026-10-30 17:48:08.432881436 +0000 UTC m=+2116.210353391" Oct 30 17:48:08.993461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.993420 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:08.994741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.994713 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 17:48:08.994894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:08.994754 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 17:48:09.993661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:09.993628 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 17:48:09.994068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:09.993685 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 17:48:10.993207 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:10.993179 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 17:48:10.993425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:10.993228 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 17:48:11.606596 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.606562 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:48:11.613824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.613796 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.636640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.636601 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-488c67\"/\"test-model-488c67-kserve-self-signed-certs\"" Oct 30 17:48:11.637357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.637336 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-488c67\"/\"default-dockercfg-qwwgn\"" Oct 30 17:48:11.637990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.637966 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:48:11.667321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.656414 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-488c67\"/\"openshift-service-ca.crt\"" Oct 30 17:48:11.667321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.656508 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-488c67\"/\"kube-root-ca.crt\"" Oct 30 17:48:11.672544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.672514 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.672693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.672591 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.672693 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.672644 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.672808 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.672797 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.672876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.672842 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gbt8\" (UniqueName: \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.673119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.673088 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775024 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.774987 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gbt8\" (UniqueName: \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775082 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775166 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775200 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775229 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775336 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.775696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.775679 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.776487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.776445 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.776620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.776539 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.778535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.778517 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.778906 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.778885 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.799068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.799046 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gbt8\" (UniqueName: \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\") pod \"test-model-488c67-kserve-748dff895-4qp9w\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:11.989336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:11.988637 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:12.001809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:12.000778 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 17:48:12.001809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:12.000832 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 17:48:12.477066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:12.477037 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:48:12.993134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:12.993096 4622 patch_prober.go:29] interesting pod/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 17:48:12.993537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:12.993154 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 17:48:13.449674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.449632 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" event={"ID":"cb4e273a-5f08-40d1-92ea-2ed2a34ac079","Type":"ContainerStarted","Data":"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d"} Oct 30 17:48:13.449674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.449675 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" event={"ID":"cb4e273a-5f08-40d1-92ea-2ed2a34ac079","Type":"ContainerStarted","Data":"5af77a12163cb95f1d746d1477e9e1da5ea5db21c606bdf0cbb9f9dfe7d05580"} Oct 30 17:48:13.499357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.499299 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" podStartSLOduration=2.499281554 podStartE2EDuration="2.499281554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:13.492346604 +0000 UTC m=+2121.269818561" watchObservedRunningTime="2026-10-30 17:48:13.499281554 +0000 UTC m=+2121.276753511" Oct 30 17:48:13.990089 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.990060 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:13.998514 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.998489 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:13.998882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.998872 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:13.999716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:13.999698 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:48:21.989338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:21.989302 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:22.001618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:22.001593 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:22.561907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:22.561875 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:48:24.535645 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:48:24.535607 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:48:24.663575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:24.663536 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:48:24.664083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:24.664046 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" containerID="cri-o://095cf7fcbed4e21ba37cc73f0d5593aa3974d0fb5f03cc3a1e13fd74986c3ee9" gracePeriod=30 Oct 30 17:48:30.449290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.448687 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:48:30.486443 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.486413 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:48:30.486760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.486740 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.514587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.514564 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-303ba5\"/\"default-dockercfg-kb4mm\"" Oct 30 17:48:30.514888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.514873 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-303ba5\"/\"openshift-service-ca.crt\"" Oct 30 17:48:30.515021 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.515011 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-303ba5\"/\"test-model-303ba5-kserve-self-signed-certs\"" Oct 30 17:48:30.515164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.515153 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-303ba5\"/\"kube-root-ca.crt\"" Oct 30 17:48:30.588276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.588231 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.588597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.588496 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvnw\" (UniqueName: \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.588809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.588787 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.589002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.588982 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.589091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.589033 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.589225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.589208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.691661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.691548 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.691931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.691911 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.692214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.692192 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.692534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.692515 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.692783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.692766 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvnw\" (UniqueName: \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.692932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.692906 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.693136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.693108 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.693900 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.693877 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.695225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.695202 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.723578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.723537 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.726109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.726081 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.732214 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.732188 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvnw\" (UniqueName: \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\") pod \"test-model-303ba5-kserve-86d9fd6c-tshk2\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.734869 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.734835 4622 generic.go:356] "Generic (PLEG): container finished" podID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerID="095cf7fcbed4e21ba37cc73f0d5593aa3974d0fb5f03cc3a1e13fd74986c3ee9" exitCode=0 Oct 30 17:48:30.734999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.734981 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" event={"ID":"53fe6e46-5076-4fe0-996a-a8d9be8359a4","Type":"ContainerDied","Data":"095cf7fcbed4e21ba37cc73f0d5593aa3974d0fb5f03cc3a1e13fd74986c3ee9"} Oct 30 17:48:30.763001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.762982 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:48:30.821361 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.821334 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:30.895739 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895700 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.895866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895829 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.895919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895879 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.895961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895928 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895981 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.895990 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.896118 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896065 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896111 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896203 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.896240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896220 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896308 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\") pod \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\" (UID: \"53fe6e46-5076-4fe0-996a-a8d9be8359a4\") " Oct 30 17:48:30.896969 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896815 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.897096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.896982 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.897385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.897347 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:30.898617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.898519 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.898617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.898546 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.898617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.898562 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.898617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.898578 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.898617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.898594 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:30.900712 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.900677 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:30.900813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.900718 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:30.900813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.900751 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "kube-api-access-nz5fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:30.902193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:30.902001 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo" pod "53fe6e46-5076-4fe0-996a-a8d9be8359a4" (UID: "53fe6e46-5076-4fe0-996a-a8d9be8359a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:48:31.000558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.000487 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:31.000558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.000513 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:31.000558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.000524 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fe6e46-5076-4fe0-996a-a8d9be8359a4-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:31.000558 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.000533 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz5fg\" (UniqueName: \"kubernetes.io/projected/53fe6e46-5076-4fe0-996a-a8d9be8359a4-kube-api-access-nz5fg\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:48:31.144647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.144617 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:48:31.145199 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:48:31.145176 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3c2f1f7_a15c_4011_bb85_65cc90fc59c0.slice/crio-5cba4c1d07b1fc4d245df363c839597807026b6c0e175d57021b206615f40360 WatchSource:0}: Error finding container 5cba4c1d07b1fc4d245df363c839597807026b6c0e175d57021b206615f40360: Status 404 returned error can't find the container with id 5cba4c1d07b1fc4d245df363c839597807026b6c0e175d57021b206615f40360 Oct 30 17:48:31.745650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.745618 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" Oct 30 17:48:31.746088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.745654 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc" event={"ID":"53fe6e46-5076-4fe0-996a-a8d9be8359a4","Type":"ContainerDied","Data":"363c61f39901df89a3c306adce9857cf904571d720654ca3f333d3d35f34a28f"} Oct 30 17:48:31.746088 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.745723 4622 scope.go:122] "RemoveContainer" containerID="095cf7fcbed4e21ba37cc73f0d5593aa3974d0fb5f03cc3a1e13fd74986c3ee9" Oct 30 17:48:31.747622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.747585 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" event={"ID":"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0","Type":"ContainerStarted","Data":"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430"} Oct 30 17:48:31.747622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.747624 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" event={"ID":"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0","Type":"ContainerStarted","Data":"5cba4c1d07b1fc4d245df363c839597807026b6c0e175d57021b206615f40360"} Oct 30 17:48:31.801463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.801439 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:48:31.812380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.812353 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bba90b-openshift-default-78c49d8b95-ddpjc"] Oct 30 17:48:31.821610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.821587 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:31.842177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:31.842140 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" podStartSLOduration=2.842129936 podStartE2EDuration="2.842129936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:48:31.8403362 +0000 UTC m=+2139.617808159" watchObservedRunningTime="2026-10-30 17:48:31.842129936 +0000 UTC m=+2139.619601890" Oct 30 17:48:32.377416 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:32.377387 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" path="/var/lib/kubelet/pods/53fe6e46-5076-4fe0-996a-a8d9be8359a4/volumes" Oct 30 17:48:40.821950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:40.821922 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:40.834774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:40.834750 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:41.833993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:41.833963 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:48:54.535668 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:48:54.535631 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:48:56.527210 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:56.522826 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:48:56.527210 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:48:56.523067 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" containerID="cri-o://605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a" gracePeriod=30 Oct 30 17:49:01.893432 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:01.893398 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:49:02.014621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.014591 4622 generic.go:356] "Generic (PLEG): container finished" podID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerID="605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a" exitCode=0 Oct 30 17:49:02.014792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.014639 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" Oct 30 17:49:02.014792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.014679 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" event={"ID":"f66f2487-4ccf-4817-9646-b23ac17e39e4","Type":"ContainerDied","Data":"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a"} Oct 30 17:49:02.014792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.014752 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz" event={"ID":"f66f2487-4ccf-4817-9646-b23ac17e39e4","Type":"ContainerDied","Data":"510c1e756f4a5d5c8fcaed3f9e4a4983b7f446df04f60afddd4f9158d4dc5a0e"} Oct 30 17:49:02.014792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.014775 4622 scope.go:122] "RemoveContainer" containerID="605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a" Oct 30 17:49:02.024327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024252 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024481 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024467 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024525 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024506 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024543 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024600 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024580 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024694 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.024820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024731 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:02.024820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024793 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:02.024820 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024822 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.025132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024863 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.025132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024914 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\") pod \"f66f2487-4ccf-4817-9646-b23ac17e39e4\" (UID: \"f66f2487-4ccf-4817-9646-b23ac17e39e4\") " Oct 30 17:49:02.025132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.024932 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:02.025132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025075 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:02.025416 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025378 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:02.025746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025720 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.025746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025746 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.025921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025762 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f66f2487-4ccf-4817-9646-b23ac17e39e4-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.025921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025778 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.025921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.025792 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.026168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.026086 4622 scope.go:122] "RemoveContainer" containerID="605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a" Oct 30 17:49:02.026481 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:02.026456 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a\": container with ID starting with 605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a not found: ID does not exist" containerID="605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a" Oct 30 17:49:02.026589 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.026493 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a"} err="failed to get container status \"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a\": rpc error: code = NotFound desc = could not find container \"605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a\": container with ID starting with 605455c290232e0634da00aabf18450fb1ee103c6b6cd89cf8fae8f70728922a not found: ID does not exist" Oct 30 17:49:02.026874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.026852 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:02.026956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.026907 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:02.027127 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.027108 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:02.027327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.027309 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx" pod "f66f2487-4ccf-4817-9646-b23ac17e39e4" (UID: "f66f2487-4ccf-4817-9646-b23ac17e39e4"). InnerVolumeSpecName "kube-api-access-jfzmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:02.127468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.127395 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzmx\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-kube-api-access-jfzmx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.127468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.127421 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.127468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.127430 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.127468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.127438 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f66f2487-4ccf-4817-9646-b23ac17e39e4-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:02.366027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.365999 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:49:02.378145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:02.378084 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-455086-openshift-default-6ffc67dd7-zn2wz"] Oct 30 17:49:04.377218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:04.377179 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" path="/var/lib/kubelet/pods/f66f2487-4ccf-4817-9646-b23ac17e39e4/volumes" Oct 30 17:49:04.617799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:04.617763 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:49:11.795306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:11.795257 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:49:11.795731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:11.795488 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" containerID="cri-o://2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3" gracePeriod=30 Oct 30 17:49:16.184282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:16.183384 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:49:16.184282 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:16.183601 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" containerID="cri-o://abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5" gracePeriod=30 Oct 30 17:49:17.089002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.088980 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:49:17.179742 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.179702 4622 generic.go:356] "Generic (PLEG): container finished" podID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerID="2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3" exitCode=0 Oct 30 17:49:17.179919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.179761 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" Oct 30 17:49:17.179919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.179790 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" event={"ID":"50fe9f14-a4be-46be-a29e-4b515fc17aa7","Type":"ContainerDied","Data":"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3"} Oct 30 17:49:17.179919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.179859 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs" event={"ID":"50fe9f14-a4be-46be-a29e-4b515fc17aa7","Type":"ContainerDied","Data":"d78c79060ec6f0dbb52597cdeb9d1cf8c6ea5dec32ed5148dc842070244b1c76"} Oct 30 17:49:17.179919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.179892 4622 scope.go:122] "RemoveContainer" containerID="2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3" Oct 30 17:49:17.191855 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.191658 4622 scope.go:122] "RemoveContainer" containerID="2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3" Oct 30 17:49:17.192141 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:17.191966 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3\": container with ID starting with 2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3 not found: ID does not exist" containerID="2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3" Oct 30 17:49:17.192141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.191993 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3"} err="failed to get container status \"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3\": rpc error: code = NotFound desc = could not find container \"2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3\": container with ID starting with 2ba8f22654957baa5b322de484184ec2a46ab00c38597a1cc4f75ddfff4fc5f3 not found: ID does not exist" Oct 30 17:49:17.208054 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208030 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208105 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208253 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208409 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208465 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208507 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208638 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208582 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208649 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208704 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\") pod \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\" (UID: \"50fe9f14-a4be-46be-a29e-4b515fc17aa7\") " Oct 30 17:49:17.208791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208580 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.208886 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.208755 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:17.209066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209043 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.209347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209321 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.209508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209480 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.209616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209562 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.209616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209602 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.209745 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209618 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.209745 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.209631 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.211128 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.211104 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "kube-api-access-qk77w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:17.211436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.211419 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:17.212228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.212201 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:17.212597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.212578 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo" pod "50fe9f14-a4be-46be-a29e-4b515fc17aa7" (UID: "50fe9f14-a4be-46be-a29e-4b515fc17aa7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:17.322359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.311130 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.322359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.311170 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.322359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.311182 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.322359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.311194 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk77w\" (UniqueName: \"kubernetes.io/projected/50fe9f14-a4be-46be-a29e-4b515fc17aa7-kube-api-access-qk77w\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.322359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.311213 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50fe9f14-a4be-46be-a29e-4b515fc17aa7-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:17.541288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.533235 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:49:17.568415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:17.568374 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-pl5cs"] Oct 30 17:49:18.186602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:18.186562 4622 patch_prober.go:29] interesting pod/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:49:18.186805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:18.186625 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:49:18.393283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:18.391356 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" path="/var/lib/kubelet/pods/50fe9f14-a4be-46be-a29e-4b515fc17aa7/volumes" Oct 30 17:49:19.648414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.648377 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" podUID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" containerName="main" containerID="cri-o://c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d" gracePeriod=45 Oct 30 17:49:19.928346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.928319 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:49:19.987073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987031 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987077 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987173 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\" (UniqueName: \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987402 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987451 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987619 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987605 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\") pod \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\" (UID: \"bf264a5d-c7ea-4fc7-855f-93fc980c8da1\") " Oct 30 17:49:19.987867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.987847 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:19.988029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.988013 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:19.988196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.988172 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:19.988802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.988781 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:19.988902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.988804 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:19.988902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.988816 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:19.991087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.991040 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:19.991202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.991114 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:19.991474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:19.991451 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp" pod "bf264a5d-c7ea-4fc7-855f-93fc980c8da1" (UID: "bf264a5d-c7ea-4fc7-855f-93fc980c8da1"). InnerVolumeSpecName "kube-api-access-5f9wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:20.092295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.092245 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9wp\" (UniqueName: \"kubernetes.io/projected/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-kube-api-access-5f9wp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:20.092295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.092295 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:20.092490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.092313 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf264a5d-c7ea-4fc7-855f-93fc980c8da1-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:20.227338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.225471 4622 generic.go:356] "Generic (PLEG): container finished" podID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" containerID="c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d" exitCode=0 Oct 30 17:49:20.227338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.225626 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" event={"ID":"bf264a5d-c7ea-4fc7-855f-93fc980c8da1","Type":"ContainerDied","Data":"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d"} Oct 30 17:49:20.227338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.225661 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" event={"ID":"bf264a5d-c7ea-4fc7-855f-93fc980c8da1","Type":"ContainerDied","Data":"f563c181cb9810cce8f871b1a9aeeb35d1498e8fe5314d58e3994893fb9f0540"} Oct 30 17:49:20.227338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.225681 4622 scope.go:122] "RemoveContainer" containerID="c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d" Oct 30 17:49:20.227338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.225813 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh" Oct 30 17:49:20.246441 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.246413 4622 scope.go:122] "RemoveContainer" containerID="c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d" Oct 30 17:49:20.246734 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:20.246712 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d\": container with ID starting with c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d not found: ID does not exist" containerID="c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d" Oct 30 17:49:20.246797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.246743 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d"} err="failed to get container status \"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d\": rpc error: code = NotFound desc = could not find container \"c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d\": container with ID starting with c9514440fc2053d269efce02dbba18bacf2e5a0d7358aba6be1e2f487d1fa50d not found: ID does not exist" Oct 30 17:49:20.284090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.284061 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:49:20.288350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.288324 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-874b83/test-model-874b83-kserve-5874fbcb8c-pm2vh"] Oct 30 17:49:20.376385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:20.376354 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" path="/var/lib/kubelet/pods/bf264a5d-c7ea-4fc7-855f-93fc980c8da1/volumes" Oct 30 17:49:21.474720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.474695 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:49:21.519581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519548 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519589 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519617 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519670 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519755 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519745 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519769 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519804 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519827 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:21.519968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519883 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.519968 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.519938 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\") pod \"4fffe92d-deb6-4993-8111-88e79bab77a7\" (UID: \"4fffe92d-deb6-4993-8111-88e79bab77a7\") " Oct 30 17:49:21.520311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520002 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:21.520311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520024 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:49:21.520311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520125 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:21.520311 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520201 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:21.520735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520705 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fffe92d-deb6-4993-8111-88e79bab77a7-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.520735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520732 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.520919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520748 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.520919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520762 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.520919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.520777 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.522348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.522321 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:49:21.522437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.522380 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:21.522610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.522584 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:21.522713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.522682 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp" pod "4fffe92d-deb6-4993-8111-88e79bab77a7" (UID: "4fffe92d-deb6-4993-8111-88e79bab77a7"). InnerVolumeSpecName "kube-api-access-d6snp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:21.622585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.622546 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.622585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.622585 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.622798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.622599 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fffe92d-deb6-4993-8111-88e79bab77a7-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:21.622798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:21.622614 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6snp\" (UniqueName: \"kubernetes.io/projected/4fffe92d-deb6-4993-8111-88e79bab77a7-kube-api-access-d6snp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:22.242827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.242790 4622 generic.go:356] "Generic (PLEG): container finished" podID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerID="abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5" exitCode=0 Oct 30 17:49:22.243005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.242824 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" event={"ID":"4fffe92d-deb6-4993-8111-88e79bab77a7","Type":"ContainerDied","Data":"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5"} Oct 30 17:49:22.243005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.242851 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" Oct 30 17:49:22.243005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.242915 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s" event={"ID":"4fffe92d-deb6-4993-8111-88e79bab77a7","Type":"ContainerDied","Data":"e981f06478b4c3266d4abaff438ebcdc8bb35c07d463efb83601dd5b8f47944e"} Oct 30 17:49:22.243005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.242951 4622 scope.go:122] "RemoveContainer" containerID="abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5" Oct 30 17:49:22.254946 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.254931 4622 scope.go:122] "RemoveContainer" containerID="abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5" Oct 30 17:49:22.255346 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:22.255224 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5\": container with ID starting with abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5 not found: ID does not exist" containerID="abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5" Oct 30 17:49:22.255346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.255278 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5"} err="failed to get container status \"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5\": rpc error: code = NotFound desc = could not find container \"abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5\": container with ID starting with abeea7bde62efc3c18f370428c6eeaee5801bab78dd1f4c10847570380b86ac5 not found: ID does not exist" Oct 30 17:49:22.305356 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.305310 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:49:22.316716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.316684 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-143041-openshift-default-5cdbb9f899-xs95s"] Oct 30 17:49:22.376449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:22.376416 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" path="/var/lib/kubelet/pods/4fffe92d-deb6-4993-8111-88e79bab77a7/volumes" Oct 30 17:49:24.535644 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:24.535612 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:49:28.477011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:28.476970 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:49:34.169861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:34.169829 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:49:35.533166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533129 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:49:35.533854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533839 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" Oct 30 17:49:35.533901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533855 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" Oct 30 17:49:35.533901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533866 4622 cpu_manager.go:414] "containerMap: removing container" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" Oct 30 17:49:35.533901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533872 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" Oct 30 17:49:35.533901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533885 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" Oct 30 17:49:35.533901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533891 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533907 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" containerName="main" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533912 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" containerName="main" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533923 4622 cpu_manager.go:414] "containerMap: removing container" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.533927 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.534038 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66f2487-4ccf-4817-9646-b23ac17e39e4" containerName="istio-proxy" Oct 30 17:49:35.534052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.534045 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="50fe9f14-a4be-46be-a29e-4b515fc17aa7" containerName="istio-proxy" Oct 30 17:49:35.534226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.534054 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fffe92d-deb6-4993-8111-88e79bab77a7" containerName="istio-proxy" Oct 30 17:49:35.534226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.534063 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf264a5d-c7ea-4fc7-855f-93fc980c8da1" containerName="main" Oct 30 17:49:35.534226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.534073 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fe6e46-5076-4fe0-996a-a8d9be8359a4" containerName="istio-proxy" Oct 30 17:49:35.538519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.538496 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.559730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.555464 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:49:35.562060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.562017 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-043253\"/\"default-dockercfg-mqt7w\"" Oct 30 17:49:35.562687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.562664 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-043253\"/\"test-model-043253-kserve-self-signed-certs\"" Oct 30 17:49:35.567150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.567120 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-043253\"/\"openshift-service-ca.crt\"" Oct 30 17:49:35.567280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.567169 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-043253\"/\"kube-root-ca.crt\"" Oct 30 17:49:35.639310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639275 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.639482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639332 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4j4d\" (UniqueName: \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.639482 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639368 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.639592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639559 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.639592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639586 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.639812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.639785 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.740981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.740940 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741004 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741156 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741221 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741252 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4j4d\" (UniqueName: \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741323 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741406 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741699 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.741963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.741941 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.743723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.743678 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.743723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.743707 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.765178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.765151 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4j4d\" (UniqueName: \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\") pod \"test-model-043253-kserve-5955984d9c-g9t4t\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:35.869563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:35.869483 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:36.262801 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:49:36.262774 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82a6ced5_347f_4dbb_92ed_6edb421a03c0.slice/crio-72a917be5d35704826d51cf5947759c17b1fe03aaa176dc0739f027591e58898 WatchSource:0}: Error finding container 72a917be5d35704826d51cf5947759c17b1fe03aaa176dc0739f027591e58898: Status 404 returned error can't find the container with id 72a917be5d35704826d51cf5947759c17b1fe03aaa176dc0739f027591e58898 Oct 30 17:49:36.263908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:36.263877 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:49:36.396293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:36.396250 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" event={"ID":"82a6ced5-347f-4dbb-92ed-6edb421a03c0","Type":"ContainerStarted","Data":"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd"} Oct 30 17:49:36.396293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:36.396301 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" event={"ID":"82a6ced5-347f-4dbb-92ed-6edb421a03c0","Type":"ContainerStarted","Data":"72a917be5d35704826d51cf5947759c17b1fe03aaa176dc0739f027591e58898"} Oct 30 17:49:36.438080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:36.438032 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" podStartSLOduration=1.4380181140000001 podStartE2EDuration="1.438018114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:36.4376636 +0000 UTC m=+2204.215135557" watchObservedRunningTime="2026-10-30 17:49:36.438018114 +0000 UTC m=+2204.215490070" Oct 30 17:49:36.870751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:36.870720 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:43.520193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.520155 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" podUID="d036cba4-d86e-4c12-84cd-541e81e99455" containerName="main" containerID="cri-o://c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477" gracePeriod=45 Oct 30 17:49:43.848771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.848742 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:49:43.981341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981294 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.981553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981534 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\" (UniqueName: \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.981630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981564 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:43.981630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981597 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.981741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981730 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\" (UniqueName: \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.981834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981822 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.981934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.981862 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\") pod \"d036cba4-d86e-4c12-84cd-541e81e99455\" (UID: \"d036cba4-d86e-4c12-84cd-541e81e99455\") " Oct 30 17:49:43.982509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.982242 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:43.982509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.982463 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:43.988077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.984386 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:43.988077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.986963 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:43.988077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.986985 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:43.988077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.987000 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:43.988077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.987012 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d036cba4-d86e-4c12-84cd-541e81e99455-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:43.993308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.993177 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:43.994785 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:43.994754 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq" pod "d036cba4-d86e-4c12-84cd-541e81e99455" (UID: "d036cba4-d86e-4c12-84cd-541e81e99455"). InnerVolumeSpecName "kube-api-access-vm8tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:44.088963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.088887 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d036cba4-d86e-4c12-84cd-541e81e99455-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:44.088963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.088918 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8tq\" (UniqueName: \"kubernetes.io/projected/d036cba4-d86e-4c12-84cd-541e81e99455-kube-api-access-vm8tq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:44.476080 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.476046 4622 generic.go:356] "Generic (PLEG): container finished" podID="d036cba4-d86e-4c12-84cd-541e81e99455" containerID="c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477" exitCode=0 Oct 30 17:49:44.476300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.476096 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" Oct 30 17:49:44.476300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.476131 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" event={"ID":"d036cba4-d86e-4c12-84cd-541e81e99455","Type":"ContainerDied","Data":"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477"} Oct 30 17:49:44.476300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.476174 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc" event={"ID":"d036cba4-d86e-4c12-84cd-541e81e99455","Type":"ContainerDied","Data":"0329c497cf99bc354e38d9f82d1111ee2509e2050c889b8d9e1e862c2fae29bd"} Oct 30 17:49:44.476300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.476196 4622 scope.go:122] "RemoveContainer" containerID="c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477" Oct 30 17:49:44.487052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.487032 4622 scope.go:122] "RemoveContainer" containerID="c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477" Oct 30 17:49:44.487351 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:44.487332 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477\": container with ID starting with c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477 not found: ID does not exist" containerID="c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477" Oct 30 17:49:44.487410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.487358 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477"} err="failed to get container status \"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477\": rpc error: code = NotFound desc = could not find container \"c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477\": container with ID starting with c00704ece30563aae9bd42024c2a43116ffc229108180e689fd105dbe8fe2477 not found: ID does not exist" Oct 30 17:49:44.589864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.589833 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:49:44.603492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:44.603463 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f8b48d/test-model-f8b48d-kserve-54dddcb5f5-fj6dc"] Oct 30 17:49:45.870746 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:45.870717 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:45.883571 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:45.883537 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:46.375869 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:46.375835 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d036cba4-d86e-4c12-84cd-541e81e99455" path="/var/lib/kubelet/pods/d036cba4-d86e-4c12-84cd-541e81e99455/volumes" Oct 30 17:49:46.586872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:46.586845 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:49:49.192932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.192892 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" podUID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" containerName="main" containerID="cri-o://7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253" gracePeriod=45 Oct 30 17:49:49.627571 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.622246 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:49:49.663291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.650715 4622 generic.go:356] "Generic (PLEG): container finished" podID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" containerID="7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253" exitCode=0 Oct 30 17:49:49.663291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.650783 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" event={"ID":"da0037bb-2076-4137-aa71-e7d7e0e4bf3d","Type":"ContainerDied","Data":"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253"} Oct 30 17:49:49.663291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.650819 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" event={"ID":"da0037bb-2076-4137-aa71-e7d7e0e4bf3d","Type":"ContainerDied","Data":"417d983b7aed6b0650da79474f553f09722f7db8c2c588a5bfef0abe89868c63"} Oct 30 17:49:49.663291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.650843 4622 scope.go:122] "RemoveContainer" containerID="7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253" Oct 30 17:49:49.663291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.651164 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q" Oct 30 17:49:49.699478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.693130 4622 scope.go:122] "RemoveContainer" containerID="7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253" Oct 30 17:49:49.712295 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:49.711804 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253\": container with ID starting with 7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253 not found: ID does not exist" containerID="7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253" Oct 30 17:49:49.712295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.711850 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253"} err="failed to get container status \"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253\": rpc error: code = NotFound desc = could not find container \"7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253\": container with ID starting with 7a5610160997aebe176be1882087626549bde4595280249440b3a12d23421253 not found: ID does not exist" Oct 30 17:49:49.914056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.878100 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\" (UniqueName: \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.914056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.878200 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.914056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.878229 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.914056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.878625 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:49.914056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.886143 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:49.934283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.930716 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "kube-api-access-88zv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.958245 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.958382 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.958503 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\") pod \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\" (UID: \"da0037bb-2076-4137-aa71-e7d7e0e4bf3d\") " Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.959187 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88zv9\" (UniqueName: \"kubernetes.io/projected/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-kube-api-access-88zv9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.959212 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:49.959707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.959227 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:49.961217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.960432 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:49.961217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.960548 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:49:49.967796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:49.967771 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs" pod "da0037bb-2076-4137-aa71-e7d7e0e4bf3d" (UID: "da0037bb-2076-4137-aa71-e7d7e0e4bf3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:49:50.060288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.060225 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.060288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.060251 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.060288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.060270 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da0037bb-2076-4137-aa71-e7d7e0e4bf3d-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:49:50.327042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.326998 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:49:50.334359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.334325 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ff3/test-model-ee9ff3-kserve-6f7f558b55-2x27q"] Oct 30 17:49:50.382577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:50.382544 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" path="/var/lib/kubelet/pods/da0037bb-2076-4137-aa71-e7d7e0e4bf3d/volumes" Oct 30 17:49:51.664956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.664927 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:49:51.665618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665601 4622 cpu_manager.go:414] "containerMap: removing container" podUID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" containerName="main" Oct 30 17:49:51.665618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665616 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" containerName="main" Oct 30 17:49:51.665753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665626 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d036cba4-d86e-4c12-84cd-541e81e99455" containerName="main" Oct 30 17:49:51.665753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665631 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d036cba4-d86e-4c12-84cd-541e81e99455" containerName="main" Oct 30 17:49:51.665872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665762 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0037bb-2076-4137-aa71-e7d7e0e4bf3d" containerName="main" Oct 30 17:49:51.665872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.665770 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d036cba4-d86e-4c12-84cd-541e81e99455" containerName="main" Oct 30 17:49:51.670849 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.670830 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.696136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.695995 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:49:51.698500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.698458 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcdac0\"/\"kube-root-ca.crt\"" Oct 30 17:49:51.704129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.704104 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bcdac0\"/\"openshift-service-ca.crt\"" Oct 30 17:49:51.704255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.704146 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcdac0\"/\"default-dockercfg-7vkp4\"" Oct 30 17:49:51.704458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.704437 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bcdac0\"/\"test-model-bcdac0-kserve-self-signed-certs\"" Oct 30 17:49:51.790108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790077 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.790314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790144 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.790314 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790279 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.790450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790355 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghz29\" (UniqueName: \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.790503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790450 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.790503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.790482 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.891827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.891791 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.891856 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892055 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892004 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892071 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892124 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892170 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghz29\" (UniqueName: \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892392 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892604 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.892896 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.892850 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.894735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.894708 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.895007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.894981 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:51.972331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:51.971329 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghz29\" (UniqueName: \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\") pod \"test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:52.003284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:52.002944 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:52.400302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:52.400234 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:49:52.400302 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:49:52.400231 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c651c3b_34d0_4d27_992d_86a10f88e5e3.slice/crio-13bcae6342406b53db873bdf52e9529e6272c01bbb3565a7d821b2bbe12a4bba WatchSource:0}: Error finding container 13bcae6342406b53db873bdf52e9529e6272c01bbb3565a7d821b2bbe12a4bba: Status 404 returned error can't find the container with id 13bcae6342406b53db873bdf52e9529e6272c01bbb3565a7d821b2bbe12a4bba Oct 30 17:49:52.677897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:52.677802 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" event={"ID":"4c651c3b-34d0-4d27-992d-86a10f88e5e3","Type":"ContainerStarted","Data":"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d"} Oct 30 17:49:52.677897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:52.677864 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" event={"ID":"4c651c3b-34d0-4d27-992d-86a10f88e5e3","Type":"ContainerStarted","Data":"13bcae6342406b53db873bdf52e9529e6272c01bbb3565a7d821b2bbe12a4bba"} Oct 30 17:49:52.723302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:52.723237 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" podStartSLOduration=1.723222721 podStartE2EDuration="1.723222721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:52.721013441 +0000 UTC m=+2220.498485420" watchObservedRunningTime="2026-10-30 17:49:52.723222721 +0000 UTC m=+2220.500694699" Oct 30 17:49:53.004116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:53.004030 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:49:54.536379 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:49:54.536342 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:49:57.940832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:57.940794 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:49:57.941729 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:49:57.941667 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" containerID="cri-o://f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c" gracePeriod=30 Oct 30 17:50:00.032759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:00.032721 4622 patch_prober.go:29] interesting pod/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:50:00.033278 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:00.032818 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:50:02.004035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:02.004002 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:50:02.016726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:02.016700 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:50:02.678338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:02.675125 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:50:02.984911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:02.984882 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:50:03.328681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.328649 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:50:03.460674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460636 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.460884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460710 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.460884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460809 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.460999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460887 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.460999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460950 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.460999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.460986 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.461168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.461053 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.461168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.461139 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.461291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.461250 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\") pod \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\" (UID: \"416812c2-aea7-4a8b-9d51-120ad7e9cf0f\") " Oct 30 17:50:03.461694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.461643 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.461848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.461821 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:03.462065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462042 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.462126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462104 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.462431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462406 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.462431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462433 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.462674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462448 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.462674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462464 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.462795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.462740 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.464694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.464664 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.464985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.464953 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "kube-api-access-hgdn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:03.479461 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.479434 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:50:03.479589 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.479430 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy" pod "416812c2-aea7-4a8b-9d51-120ad7e9cf0f" (UID: "416812c2-aea7-4a8b-9d51-120ad7e9cf0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:03.563579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.563501 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.563579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.563530 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.563579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.563541 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.563579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.563550 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgdn6\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-kube-api-access-hgdn6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.563579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.563560 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/416812c2-aea7-4a8b-9d51-120ad7e9cf0f-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:03.803040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.803008 4622 generic.go:356] "Generic (PLEG): container finished" podID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerID="f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c" exitCode=0 Oct 30 17:50:03.803206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.803068 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" Oct 30 17:50:03.803206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.803097 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" event={"ID":"416812c2-aea7-4a8b-9d51-120ad7e9cf0f","Type":"ContainerDied","Data":"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c"} Oct 30 17:50:03.803206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.803151 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl" event={"ID":"416812c2-aea7-4a8b-9d51-120ad7e9cf0f","Type":"ContainerDied","Data":"1d4ba605e95a2c9eb101def39e299fdc2a4fe13e49308c140a1006e08383c91f"} Oct 30 17:50:03.803206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.803181 4622 scope.go:122] "RemoveContainer" containerID="f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c" Oct 30 17:50:03.816324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.816306 4622 scope.go:122] "RemoveContainer" containerID="f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c" Oct 30 17:50:03.816615 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:50:03.816594 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c\": container with ID starting with f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c not found: ID does not exist" containerID="f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c" Oct 30 17:50:03.816679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.816623 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c"} err="failed to get container status \"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c\": rpc error: code = NotFound desc = could not find container \"f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c\": container with ID starting with f39ffe789acc11da0ac92b94eae737d72eed85286eebbaf94390a477fb97d77c not found: ID does not exist" Oct 30 17:50:03.857425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.857393 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:50:03.869668 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:03.869643 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-874b83-openshift-default-7db5cb98c-w6mfl"] Oct 30 17:50:04.376655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:04.376623 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" path="/var/lib/kubelet/pods/416812c2-aea7-4a8b-9d51-120ad7e9cf0f/volumes" Oct 30 17:50:17.709829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:17.709761 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" podUID="29177b02-a141-4690-8f91-7978b2721f45" containerName="main" containerID="cri-o://d403f3e0631fd29c177e44973286dda599115f61cef8b58ac894c4642e1a3d9b" gracePeriod=45 Oct 30 17:50:18.104160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.101438 4622 generic.go:356] "Generic (PLEG): container finished" podID="29177b02-a141-4690-8f91-7978b2721f45" containerID="d403f3e0631fd29c177e44973286dda599115f61cef8b58ac894c4642e1a3d9b" exitCode=0 Oct 30 17:50:18.104160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.101683 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" event={"ID":"29177b02-a141-4690-8f91-7978b2721f45","Type":"ContainerDied","Data":"d403f3e0631fd29c177e44973286dda599115f61cef8b58ac894c4642e1a3d9b"} Oct 30 17:50:18.143283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.139442 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:50:18.261174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261139 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\" (UniqueName: \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.261399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261237 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.261399 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261364 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\" (UniqueName: \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.261526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261431 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.261592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261536 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:18.261706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261684 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:18.261800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261785 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.261932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.261913 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\") pod \"29177b02-a141-4690-8f91-7978b2721f45\" (UID: \"29177b02-a141-4690-8f91-7978b2721f45\") " Oct 30 17:50:18.262067 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.262031 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:18.262730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.262705 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:18.262730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.262732 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:18.262992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.262745 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:18.263834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.263810 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:18.264017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.263990 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "kube-api-access-mtt48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:18.264107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.264064 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm" pod "29177b02-a141-4690-8f91-7978b2721f45" (UID: "29177b02-a141-4690-8f91-7978b2721f45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:18.364186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.364115 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29177b02-a141-4690-8f91-7978b2721f45-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:18.364186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.364141 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29177b02-a141-4690-8f91-7978b2721f45-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:18.364186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:18.364152 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtt48\" (UniqueName: \"kubernetes.io/projected/29177b02-a141-4690-8f91-7978b2721f45-kube-api-access-mtt48\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:19.110557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:19.110526 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" Oct 30 17:50:19.110557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:19.110544 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf" event={"ID":"29177b02-a141-4690-8f91-7978b2721f45","Type":"ContainerDied","Data":"8b0b0af593adb10a798b0619be15cd696c1a39585583caef77f39249536a5d5d"} Oct 30 17:50:19.111020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:19.110597 4622 scope.go:122] "RemoveContainer" containerID="d403f3e0631fd29c177e44973286dda599115f61cef8b58ac894c4642e1a3d9b" Oct 30 17:50:19.165919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:19.165894 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:50:19.175942 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:19.175907 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24fb0d/test-model-24fb0d-kserve-9cc68795f-4v7kf"] Oct 30 17:50:20.375586 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:20.375554 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29177b02-a141-4690-8f91-7978b2721f45" path="/var/lib/kubelet/pods/29177b02-a141-4690-8f91-7978b2721f45/volumes" Oct 30 17:50:24.539166 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:50:24.539107 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:50:40.963307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:40.963225 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:50:40.963919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:40.963512 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerName="maas-api" containerID="cri-o://ded6f404d3d76e957c31cc0950ab41de6b3c96a7dc9027bbbc60f78b374421b9" gracePeriod=30 Oct 30 17:50:41.411617 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.411577 4622 generic.go:356] "Generic (PLEG): container finished" podID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerID="ded6f404d3d76e957c31cc0950ab41de6b3c96a7dc9027bbbc60f78b374421b9" exitCode=0 Oct 30 17:50:41.411826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.411732 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" event={"ID":"16d01767-d84c-4869-81b5-b0581bfb4de1","Type":"ContainerDied","Data":"ded6f404d3d76e957c31cc0950ab41de6b3c96a7dc9027bbbc60f78b374421b9"} Oct 30 17:50:41.411826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.411783 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" event={"ID":"16d01767-d84c-4869-81b5-b0581bfb4de1","Type":"ContainerDied","Data":"453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff"} Oct 30 17:50:41.411826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.411807 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453a9161e5de6f8b144783b47db448f513d2417f5a8d990415522d5f2092bcff" Oct 30 17:50:41.426975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.426905 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:50:41.521824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.521790 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\" (UniqueName: \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\") pod \"16d01767-d84c-4869-81b5-b0581bfb4de1\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " Oct 30 17:50:41.522144 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.522123 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\") pod \"16d01767-d84c-4869-81b5-b0581bfb4de1\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " Oct 30 17:50:41.522371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.522352 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\") pod \"16d01767-d84c-4869-81b5-b0581bfb4de1\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " Oct 30 17:50:41.522721 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.522705 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\") pod \"16d01767-d84c-4869-81b5-b0581bfb4de1\" (UID: \"16d01767-d84c-4869-81b5-b0581bfb4de1\") " Oct 30 17:50:41.532077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.526328 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca" pod "16d01767-d84c-4869-81b5-b0581bfb4de1" (UID: "16d01767-d84c-4869-81b5-b0581bfb4de1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:41.563366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.563319 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc" pod "16d01767-d84c-4869-81b5-b0581bfb4de1" (UID: "16d01767-d84c-4869-81b5-b0581bfb4de1"). InnerVolumeSpecName "kube-api-access-5mcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:41.591544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.591460 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls" pod "16d01767-d84c-4869-81b5-b0581bfb4de1" (UID: "16d01767-d84c-4869-81b5-b0581bfb4de1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:41.605801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.605759 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs" pod "16d01767-d84c-4869-81b5-b0581bfb4de1" (UID: "16d01767-d84c-4869-81b5-b0581bfb4de1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:50:41.653414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.645279 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:41.653414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.645316 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcjc\" (UniqueName: \"kubernetes.io/projected/16d01767-d84c-4869-81b5-b0581bfb4de1-kube-api-access-5mcjc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:41.653414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.645332 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16d01767-d84c-4869-81b5-b0581bfb4de1-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:41.653414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:41.645346 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16d01767-d84c-4869-81b5-b0581bfb4de1-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:42.418736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:42.418707 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" Oct 30 17:50:42.479509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:42.479474 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:50:42.491952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:42.491918 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx"] Oct 30 17:50:44.185992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:44.185943 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ee9ff3-66bc456644-4csfx" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.127:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:50:44.375588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:44.375555 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" path="/var/lib/kubelet/pods/16d01767-d84c-4869-81b5-b0581bfb4de1/volumes" Oct 30 17:50:46.581273 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:46.581208 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:50:46.582046 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:46.581492 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" containerID="cri-o://f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43" gracePeriod=30 Oct 30 17:50:51.898794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:51.898767 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:50:52.033410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033373 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033432 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033530 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033600 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033640 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033678 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033719 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:52.033921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033883 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.033975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033920 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.034027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033971 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:52.034027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033986 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\") pod \"7604ad68-9342-45d8-be4d-4af284543004\" (UID: \"7604ad68-9342-45d8-be4d-4af284543004\") " Oct 30 17:50:52.034027 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.033996 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:50:52.034250 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034211 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:52.034364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034331 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:52.034779 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034755 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.034875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034784 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.034875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034801 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.034875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034815 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7604ad68-9342-45d8-be4d-4af284543004-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.034875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.034830 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.036134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.036105 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:52.036228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.036160 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:50:52.036228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.036194 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:50:52.036353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.036323 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t" pod "7604ad68-9342-45d8-be4d-4af284543004" (UID: "7604ad68-9342-45d8-be4d-4af284543004"). InnerVolumeSpecName "kube-api-access-g989t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:50:52.136685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.136587 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g989t\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-kube-api-access-g989t\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.136685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.136622 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7604ad68-9342-45d8-be4d-4af284543004-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.136685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.136635 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7604ad68-9342-45d8-be4d-4af284543004-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.136685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.136645 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7604ad68-9342-45d8-be4d-4af284543004-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:50:52.496744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.496624 4622 generic.go:356] "Generic (PLEG): container finished" podID="7604ad68-9342-45d8-be4d-4af284543004" containerID="f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43" exitCode=0 Oct 30 17:50:52.496744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.496678 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerDied","Data":"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43"} Oct 30 17:50:52.496744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.496698 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" Oct 30 17:50:52.496744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.496733 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd" event={"ID":"7604ad68-9342-45d8-be4d-4af284543004","Type":"ContainerDied","Data":"d9e3aa1cfc1d22650fff714de2374a1308ae192517cddf7da995332d0d6083c7"} Oct 30 17:50:52.497084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.496765 4622 scope.go:122] "RemoveContainer" containerID="f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43" Oct 30 17:50:52.507662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.507641 4622 scope.go:122] "RemoveContainer" containerID="dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da" Oct 30 17:50:52.520068 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.520044 4622 scope.go:122] "RemoveContainer" containerID="f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43" Oct 30 17:50:52.520511 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:50:52.520490 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43\": container with ID starting with f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43 not found: ID does not exist" containerID="f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43" Oct 30 17:50:52.520601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.520522 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43"} err="failed to get container status \"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43\": rpc error: code = NotFound desc = could not find container \"f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43\": container with ID starting with f9aac5b9a2cb682cf0966a9c87a09dd5a94cb8b2d8252c547904834c762a1c43 not found: ID does not exist" Oct 30 17:50:52.520601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.520543 4622 scope.go:122] "RemoveContainer" containerID="dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da" Oct 30 17:50:52.520823 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:50:52.520804 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da\": container with ID starting with dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da not found: ID does not exist" containerID="dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da" Oct 30 17:50:52.520883 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.520831 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da"} err="failed to get container status \"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da\": rpc error: code = NotFound desc = could not find container \"dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da\": container with ID starting with dc01db9d580d452f17fc9a183a696f2ec754946419a8c8cca7eb54414e8c52da not found: ID does not exist" Oct 30 17:50:52.563122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.563090 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:50:52.575404 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:52.575373 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f8b48d-openshift-default-7d7d8d88c5-nmvpd"] Oct 30 17:50:54.371037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:54.371002 4622 scope.go:122] "RemoveContainer" containerID="ded6f404d3d76e957c31cc0950ab41de6b3c96a7dc9027bbbc60f78b374421b9" Oct 30 17:50:54.378960 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:50:54.378925 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7604ad68-9342-45d8-be4d-4af284543004" path="/var/lib/kubelet/pods/7604ad68-9342-45d8-be4d-4af284543004/volumes" Oct 30 17:50:54.535376 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:50:54.535340 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:51:10.139064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:10.139029 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:51:24.535881 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:24.535844 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:51:24.991499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:24.988034 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:51:25.179649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.179463 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerName="main" containerID="cri-o://b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace" gracePeriod=45 Oct 30 17:51:25.553099 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.553067 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:51:25.650929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.650894 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.651120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.650958 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.651120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.651051 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.651120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.651119 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.651357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.651184 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.651357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.651238 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\" (UniqueName: \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\") pod \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\" (UID: \"3ed8eca0-ac15-44b3-9d77-2352d1204f3a\") " Oct 30 17:51:25.652429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.652370 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:25.653592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.652961 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:25.653592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.653151 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:25.655736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.655708 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "kube-api-access-p9ndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:25.656106 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.656078 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:25.657107 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.657089 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs" pod "3ed8eca0-ac15-44b3-9d77-2352d1204f3a" (UID: "3ed8eca0-ac15-44b3-9d77-2352d1204f3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:25.753379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753345 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.753379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753373 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.753379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753387 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.753637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753422 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.753637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753432 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.753637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.753444 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ndq\" (UniqueName: \"kubernetes.io/projected/3ed8eca0-ac15-44b3-9d77-2352d1204f3a-kube-api-access-p9ndq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.781636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.781544 4622 generic.go:356] "Generic (PLEG): container finished" podID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerID="b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace" exitCode=0 Oct 30 17:51:25.781636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.781599 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" Oct 30 17:51:25.781636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.781626 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" event={"ID":"3ed8eca0-ac15-44b3-9d77-2352d1204f3a","Type":"ContainerDied","Data":"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace"} Oct 30 17:51:25.781996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.781682 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" event={"ID":"3ed8eca0-ac15-44b3-9d77-2352d1204f3a","Type":"ContainerDied","Data":"ff422c0eff2f741005f9c90204731987d07e390b51f0d1a1f5d3921735bdf450"} Oct 30 17:51:25.781996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.781704 4622 scope.go:122] "RemoveContainer" containerID="b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace" Oct 30 17:51:25.795518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.795487 4622 scope.go:122] "RemoveContainer" containerID="b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace" Oct 30 17:51:25.795802 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:25.795778 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace\": container with ID starting with b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace not found: ID does not exist" containerID="b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace" Oct 30 17:51:25.795880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.795812 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace"} err="failed to get container status \"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace\": rpc error: code = NotFound desc = could not find container \"b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace\": container with ID starting with b2d0ba97d45fb3fc9a85a3adc2331cdafb91104d0a31c82283a9623a64f22ace not found: ID does not exist" Oct 30 17:51:25.839246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.839192 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:51:25.844226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:25.844203 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx"] Oct 30 17:51:26.374987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:26.374957 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" path="/var/lib/kubelet/pods/3ed8eca0-ac15-44b3-9d77-2352d1204f3a/volumes" Oct 30 17:51:26.442621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:26.442581 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b1c345/test-model-b1c345-kserve-78cf8cd967-gp8hx" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:51:31.719102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:31.719064 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:51:31.719562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:31.719287 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" containerID="cri-o://0e5f912147a1cdb083c819c455284ccf2839caef18012fb4db0964af24354d21" gracePeriod=30 Oct 30 17:51:35.702660 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:35.702622 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-fp8cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:51:35.703141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:35.702686 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:51:36.879594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:36.879563 4622 generic.go:356] "Generic (PLEG): container finished" podID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerID="0e5f912147a1cdb083c819c455284ccf2839caef18012fb4db0964af24354d21" exitCode=0 Oct 30 17:51:36.879956 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:36.879607 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" event={"ID":"2aef45da-4bf0-42b1-a5de-800f71aa1711","Type":"ContainerDied","Data":"0e5f912147a1cdb083c819c455284ccf2839caef18012fb4db0964af24354d21"} Oct 30 17:51:36.996366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:36.996304 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:51:37.041380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041353 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041409 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041433 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041453 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041488 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041522 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041513 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041679 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041795 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041743 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.041925 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041833 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\") pod \"2aef45da-4bf0-42b1-a5de-800f71aa1711\" (UID: \"2aef45da-4bf0-42b1-a5de-800f71aa1711\") " Oct 30 17:51:37.042009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041910 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:37.042074 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.041994 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:37.042227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042174 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:37.042393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042298 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:37.042393 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042318 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:37.042666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042641 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.042734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042672 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aef45da-4bf0-42b1-a5de-800f71aa1711-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.042734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042685 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.042734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042698 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.042734 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.042712 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.043836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.043817 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "kube-api-access-72skj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:37.043923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.043862 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:37.044052 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.044037 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:37.044123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.044107 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo" pod "2aef45da-4bf0-42b1-a5de-800f71aa1711" (UID: "2aef45da-4bf0-42b1-a5de-800f71aa1711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:37.143987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.143949 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72skj\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-kube-api-access-72skj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.143987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.143983 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.143987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.143993 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.144325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.144001 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aef45da-4bf0-42b1-a5de-800f71aa1711-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:37.888494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.888453 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" event={"ID":"2aef45da-4bf0-42b1-a5de-800f71aa1711","Type":"ContainerDied","Data":"1f7a65b030188bbac078b45941f2dfc5768e73003a71a89b0af496dc7073ed35"} Oct 30 17:51:37.888494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.888493 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd" Oct 30 17:51:37.889007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.888562 4622 scope.go:122] "RemoveContainer" containerID="0e5f912147a1cdb083c819c455284ccf2839caef18012fb4db0964af24354d21" Oct 30 17:51:37.959767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.959734 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:51:37.971818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:37.971792 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-fp8cd"] Oct 30 17:51:38.193168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:38.193056 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:51:38.193454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:38.193374 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" containerID="cri-o://fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96" gracePeriod=30 Oct 30 17:51:38.421300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:38.405140 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" path="/var/lib/kubelet/pods/2aef45da-4bf0-42b1-a5de-800f71aa1711/volumes" Oct 30 17:51:38.531083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:38.531002 4622 patch_prober.go:29] interesting pod/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:51:38.531235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:38.531084 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:51:40.034062 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.034003 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" podUID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" containerName="main" containerID="cri-o://384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d" gracePeriod=45 Oct 30 17:51:40.346005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.345980 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:51:40.400280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400235 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.400431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400367 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.400431 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400425 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.400553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400469 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.400553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400541 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.400672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400544 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:40.400672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400640 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:40.400788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.400740 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\" (UniqueName: \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\") pod \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\" (UID: \"cb4e273a-5f08-40d1-92ea-2ed2a34ac079\") " Oct 30 17:51:40.401564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.401069 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:40.401564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.401498 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.401564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.401523 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.401564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.401537 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.404300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.403429 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "kube-api-access-9gbt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:40.404300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.403693 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:40.404300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.403750 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs" pod "cb4e273a-5f08-40d1-92ea-2ed2a34ac079" (UID: "cb4e273a-5f08-40d1-92ea-2ed2a34ac079"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:40.503159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.503126 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.503159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.503156 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.503159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.503168 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gbt8\" (UniqueName: \"kubernetes.io/projected/cb4e273a-5f08-40d1-92ea-2ed2a34ac079-kube-api-access-9gbt8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.928787 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.928754 4622 generic.go:356] "Generic (PLEG): container finished" podID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" containerID="384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d" exitCode=0 Oct 30 17:51:40.928961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.928802 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" Oct 30 17:51:40.928961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.928810 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" event={"ID":"cb4e273a-5f08-40d1-92ea-2ed2a34ac079","Type":"ContainerDied","Data":"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d"} Oct 30 17:51:40.928961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.928842 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w" event={"ID":"cb4e273a-5f08-40d1-92ea-2ed2a34ac079","Type":"ContainerDied","Data":"5af77a12163cb95f1d746d1477e9e1da5ea5db21c606bdf0cbb9f9dfe7d05580"} Oct 30 17:51:40.928961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.928860 4622 scope.go:122] "RemoveContainer" containerID="384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d" Oct 30 17:51:40.939916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.939899 4622 scope.go:122] "RemoveContainer" containerID="384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d" Oct 30 17:51:40.940222 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:40.940204 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d\": container with ID starting with 384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d not found: ID does not exist" containerID="384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d" Oct 30 17:51:40.940292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.940231 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d"} err="failed to get container status \"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d\": rpc error: code = NotFound desc = could not find container \"384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d\": container with ID starting with 384868a5837107b45fab460d61efcde5f26ff1964ba21d3c8b8c5b1dbb5c2e0d not found: ID does not exist" Oct 30 17:51:40.999410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:40.994877 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:51:41.011947 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:41.011900 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-488c67/test-model-488c67-kserve-748dff895-4qp9w"] Oct 30 17:51:42.376516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:42.376488 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" path="/var/lib/kubelet/pods/cb4e273a-5f08-40d1-92ea-2ed2a34ac079/volumes" Oct 30 17:51:43.643703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.643675 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:51:43.704794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.704764 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.704948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.704806 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.704948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.704888 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.705081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.704969 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.705081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705064 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.705189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705094 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.705189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705128 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.705189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705136 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:43.705189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705179 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705281 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705340 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\") pod \"1a116870-e9f7-41d5-98cc-828575727fd8\" (UID: \"1a116870-e9f7-41d5-98cc-828575727fd8\") " Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705664 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705832 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.705912 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.706170 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.706193 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.706207 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.706221 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a116870-e9f7-41d5-98cc-828575727fd8-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.706520 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.706233 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.707857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.707626 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:43.707975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.707858 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:43.707975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.707908 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "kube-api-access-j2mb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:43.708610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.708590 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo" pod "1a116870-e9f7-41d5-98cc-828575727fd8" (UID: "1a116870-e9f7-41d5-98cc-828575727fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:43.807415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.807388 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.807415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.807412 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a116870-e9f7-41d5-98cc-828575727fd8-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.807608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.807422 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2mb6\" (UniqueName: \"kubernetes.io/projected/1a116870-e9f7-41d5-98cc-828575727fd8-kube-api-access-j2mb6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.807608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.807431 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a116870-e9f7-41d5-98cc-828575727fd8-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:43.999673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.999577 4622 generic.go:356] "Generic (PLEG): container finished" podID="1a116870-e9f7-41d5-98cc-828575727fd8" containerID="fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96" exitCode=0 Oct 30 17:51:43.999673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.999656 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" Oct 30 17:51:43.999673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.999665 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerDied","Data":"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96"} Oct 30 17:51:44.000042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.999733 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6" event={"ID":"1a116870-e9f7-41d5-98cc-828575727fd8","Type":"ContainerDied","Data":"96c323ca67bfb41034e02d608bf623cf5ddb71186fb25a9cb973db0472880e23"} Oct 30 17:51:44.000042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:43.999767 4622 scope.go:122] "RemoveContainer" containerID="fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96" Oct 30 17:51:44.012580 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.012564 4622 scope.go:122] "RemoveContainer" containerID="f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d" Oct 30 17:51:44.023309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.023289 4622 scope.go:122] "RemoveContainer" containerID="fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96" Oct 30 17:51:44.023565 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:44.023542 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96\": container with ID starting with fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96 not found: ID does not exist" containerID="fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96" Oct 30 17:51:44.023640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.023573 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96"} err="failed to get container status \"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96\": rpc error: code = NotFound desc = could not find container \"fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96\": container with ID starting with fc96f276883e6a1709cdcf627fe3a634b560d837a3758779797b528bf0b09e96 not found: ID does not exist" Oct 30 17:51:44.023640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.023595 4622 scope.go:122] "RemoveContainer" containerID="f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d" Oct 30 17:51:44.023839 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:44.023817 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d\": container with ID starting with f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d not found: ID does not exist" containerID="f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d" Oct 30 17:51:44.023890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.023852 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d"} err="failed to get container status \"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d\": rpc error: code = NotFound desc = could not find container \"f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d\": container with ID starting with f8081dfc2c76b1159f91d8b749ccce38b9a9d90aa535e9bd3564481e2e2ccb3d not found: ID does not exist" Oct 30 17:51:44.057564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.057541 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:51:44.069797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.069774 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24fb0d-openshift-default-5c96dc78bf-w2vq6"] Oct 30 17:51:44.379487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:44.379451 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" path="/var/lib/kubelet/pods/1a116870-e9f7-41d5-98cc-828575727fd8/volumes" Oct 30 17:51:51.166509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:51.166476 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:51:51.166993 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:51.166730 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" containerID="cri-o://bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1" gracePeriod=30 Oct 30 17:51:54.534901 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:54.534866 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:51:55.270625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:55.270588 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:51:56.438035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.437979 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:51:56.551321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551276 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551344 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551393 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551419 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551472 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551620 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551692 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551744 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.551844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551665 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.551844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551727 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.551844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551660 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:56.551844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551799 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\") pod \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\" (UID: \"8e380bc6-eabc-4c06-8a34-ce1e5919c599\") " Oct 30 17:51:56.552102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551880 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.552102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.551984 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.552486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.552460 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.552632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.552491 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.552632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.552507 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.552632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.552522 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.552632 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.552535 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.554007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.553857 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:56.554007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.553930 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:56.554007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.553995 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:56.554325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.554302 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh" pod "8e380bc6-eabc-4c06-8a34-ce1e5919c599" (UID: "8e380bc6-eabc-4c06-8a34-ce1e5919c599"). InnerVolumeSpecName "kube-api-access-x9qmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:56.654188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.654166 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.654188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.654189 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.654347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.654198 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e380bc6-eabc-4c06-8a34-ce1e5919c599-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:56.654347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:56.654208 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9qmh\" (UniqueName: \"kubernetes.io/projected/8e380bc6-eabc-4c06-8a34-ce1e5919c599-kube-api-access-x9qmh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:51:57.096648 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.096611 4622 generic.go:356] "Generic (PLEG): container finished" podID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerID="bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1" exitCode=0 Oct 30 17:51:57.096830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.096664 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" Oct 30 17:51:57.096830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.096705 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerDied","Data":"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1"} Oct 30 17:51:57.096830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.096766 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9" event={"ID":"8e380bc6-eabc-4c06-8a34-ce1e5919c599","Type":"ContainerDied","Data":"9ad575c4dbde9846590f293151d9d3fb61beb1aac42fef89d1d843670135ce81"} Oct 30 17:51:57.096830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.096796 4622 scope.go:122] "RemoveContainer" containerID="bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1" Oct 30 17:51:57.108385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.108364 4622 scope.go:122] "RemoveContainer" containerID="51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808" Oct 30 17:51:57.118019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.118002 4622 scope.go:122] "RemoveContainer" containerID="bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1" Oct 30 17:51:57.118289 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:57.118252 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1\": container with ID starting with bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1 not found: ID does not exist" containerID="bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1" Oct 30 17:51:57.118418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.118293 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1"} err="failed to get container status \"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1\": rpc error: code = NotFound desc = could not find container \"bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1\": container with ID starting with bf6da7c92e993d5a9acf5bfe5b86542a7128ff5dcca7bf7cc3598b31c09c3af1 not found: ID does not exist" Oct 30 17:51:57.118418 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.118311 4622 scope.go:122] "RemoveContainer" containerID="51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808" Oct 30 17:51:57.118542 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:51:57.118513 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808\": container with ID starting with 51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808 not found: ID does not exist" containerID="51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808" Oct 30 17:51:57.118542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.118530 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808"} err="failed to get container status \"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808\": rpc error: code = NotFound desc = could not find container \"51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808\": container with ID starting with 51bb632a4b6cec9204fa4885373644368f3b4d69ec8af9989ef99f12a17a5808 not found: ID does not exist" Oct 30 17:51:57.152690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.152663 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:51:57.164443 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:57.164421 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-38f445-openshift-default-598c59fdf9-f69j9"] Oct 30 17:51:58.142659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.142624 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:51:58.143567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143542 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:51:58.143567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143569 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143586 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143596 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143620 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143628 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143655 4622 cpu_manager.go:414] "containerMap: removing container" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerName="main" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143662 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerName="main" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143680 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143688 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143701 4622 cpu_manager.go:414] "containerMap: removing container" podUID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" containerName="main" Oct 30 17:51:58.143736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143728 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143743 4622 cpu_manager.go:414] "containerMap: removing container" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerName="maas-api" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143751 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerName="maas-api" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143775 4622 cpu_manager.go:414] "containerMap: removing container" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143783 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143793 4622 cpu_manager.go:414] "containerMap: removing container" podUID="29177b02-a141-4690-8f91-7978b2721f45" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143800 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29177b02-a141-4690-8f91-7978b2721f45" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143834 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143841 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.143981 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="29177b02-a141-4690-8f91-7978b2721f45" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144001 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aef45da-4bf0-42b1-a5de-800f71aa1711" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144011 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144020 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144031 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144043 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed8eca0-ac15-44b3-9d77-2352d1204f3a" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144054 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4e273a-5f08-40d1-92ea-2ed2a34ac079" containerName="main" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144062 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d01767-d84c-4869-81b5-b0581bfb4de1" containerName="maas-api" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144072 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:51:58.144280 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.144084 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="416812c2-aea7-4a8b-9d51-120ad7e9cf0f" containerName="istio-proxy" Oct 30 17:51:58.149796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.149774 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.165679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.165660 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"default-dockercfg-69tbc\"" Oct 30 17:51:58.179210 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.179187 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"auth-test-model-d53ec1-kserve-self-signed-certs\"" Oct 30 17:51:58.179354 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.179330 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"openshift-service-ca.crt\"" Oct 30 17:51:58.180901 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.180880 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"kube-root-ca.crt\"" Oct 30 17:51:58.183389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.183372 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:51:58.275185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275153 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.275402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275233 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zj2c\" (UniqueName: \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.275402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275311 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.275626 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275605 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.275692 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275673 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.275732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.275710 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376433 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376446 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" path="/var/lib/kubelet/pods/8e380bc6-eabc-4c06-8a34-ce1e5919c599/volumes" Oct 30 17:51:58.376724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376481 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376507 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376549 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376605 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zj2c\" (UniqueName: \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.376724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.376639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.377174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.377151 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.377291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.377222 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.379465 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.379433 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.379631 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.379574 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.401364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.400850 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.408854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.407576 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zj2c\" (UniqueName: \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\") pod \"auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.469286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.465544 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:51:58.849226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:58.849136 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:51:58.851957 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:51:58.851927 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51418bab_0a0b_4bc7_a08f_1ac09ee8baf7.slice/crio-2c5efd929b0f31569ad39d51aa468819947789924e36fcee3a8f88725c5a526b WatchSource:0}: Error finding container 2c5efd929b0f31569ad39d51aa468819947789924e36fcee3a8f88725c5a526b: Status 404 returned error can't find the container with id 2c5efd929b0f31569ad39d51aa468819947789924e36fcee3a8f88725c5a526b Oct 30 17:51:59.115234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:59.115150 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" event={"ID":"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7","Type":"ContainerStarted","Data":"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a"} Oct 30 17:51:59.115234 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:59.115200 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" event={"ID":"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7","Type":"ContainerStarted","Data":"2c5efd929b0f31569ad39d51aa468819947789924e36fcee3a8f88725c5a526b"} Oct 30 17:51:59.160101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:59.160046 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" podStartSLOduration=1.160032432 podStartE2EDuration="1.160032432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:59.156579575 +0000 UTC m=+2346.934051530" watchObservedRunningTime="2026-10-30 17:51:59.160032432 +0000 UTC m=+2346.937504387" Oct 30 17:51:59.466040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:51:59.465966 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:52:08.466302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:08.466243 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:52:08.479433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:08.479400 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:52:09.246047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:09.246021 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:52:10.296950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.296912 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" podUID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" containerName="main" containerID="cri-o://1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430" gracePeriod=45 Oct 30 17:52:10.574121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.574094 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:52:10.702087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702054 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702221 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\" (UniqueName: \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702332 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:10.702419 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702393 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702474 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702461 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702520 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702598 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702583 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\") pod \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\" (UID: \"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0\") " Oct 30 17:52:10.702698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702679 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:10.702789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.702767 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:10.703361 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.703336 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:10.703545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.703517 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:10.703670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.703548 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:10.705686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.705663 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:10.705772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.705712 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "kube-api-access-vrvnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:10.706364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.706344 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm" pod "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" (UID: "c3c2f1f7-a15c-4011-bb85-65cc90fc59c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:10.805053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.804960 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvnw\" (UniqueName: \"kubernetes.io/projected/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-kube-api-access-vrvnw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:10.805053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.804995 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:10.805053 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:10.805009 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:11.252186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.252111 4622 generic.go:356] "Generic (PLEG): container finished" podID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" containerID="1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430" exitCode=0 Oct 30 17:52:11.252186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.252147 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" Oct 30 17:52:11.252186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.252174 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" event={"ID":"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0","Type":"ContainerDied","Data":"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430"} Oct 30 17:52:11.252446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.252209 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2" event={"ID":"c3c2f1f7-a15c-4011-bb85-65cc90fc59c0","Type":"ContainerDied","Data":"5cba4c1d07b1fc4d245df363c839597807026b6c0e175d57021b206615f40360"} Oct 30 17:52:11.252446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.252225 4622 scope.go:122] "RemoveContainer" containerID="1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430" Oct 30 17:52:11.263076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.263059 4622 scope.go:122] "RemoveContainer" containerID="1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430" Oct 30 17:52:11.263381 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:52:11.263361 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430\": container with ID starting with 1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430 not found: ID does not exist" containerID="1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430" Oct 30 17:52:11.263469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.263404 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430"} err="failed to get container status \"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430\": rpc error: code = NotFound desc = could not find container \"1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430\": container with ID starting with 1805672a75a24201305f716b3e71e7645a31e33e781bc3e1f27fbdf9e1713430 not found: ID does not exist" Oct 30 17:52:11.310050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.310011 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:52:11.318305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:11.318280 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-303ba5/test-model-303ba5-kserve-86d9fd6c-tshk2"] Oct 30 17:52:12.375365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:12.375333 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" path="/var/lib/kubelet/pods/c3c2f1f7-a15c-4011-bb85-65cc90fc59c0/volumes" Oct 30 17:52:21.919637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:21.919417 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:52:21.920129 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:21.919695 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" podUID="538963a2-cf66-4269-a2bc-30a8efb9b205" containerName="maas-api" containerID="cri-o://0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866" gracePeriod=30 Oct 30 17:52:22.321326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.316285 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:52:22.345878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.345834 4622 generic.go:356] "Generic (PLEG): container finished" podID="538963a2-cf66-4269-a2bc-30a8efb9b205" containerID="0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866" exitCode=0 Oct 30 17:52:22.346056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.345907 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" event={"ID":"538963a2-cf66-4269-a2bc-30a8efb9b205","Type":"ContainerDied","Data":"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866"} Oct 30 17:52:22.346056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.345958 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" event={"ID":"538963a2-cf66-4269-a2bc-30a8efb9b205","Type":"ContainerDied","Data":"00637c1349d597a59ea485902d2efab3c4c840c2c38e6d5bb43bb7c472fa58f4"} Oct 30 17:52:22.346056 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.345989 4622 scope.go:122] "RemoveContainer" containerID="0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866" Oct 30 17:52:22.346164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.346150 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g" Oct 30 17:52:22.363105 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.362996 4622 scope.go:122] "RemoveContainer" containerID="0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866" Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:52:22.368439 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866\": container with ID starting with 0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866 not found: ID does not exist" containerID="0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866" Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.368487 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866"} err="failed to get container status \"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866\": rpc error: code = NotFound desc = could not find container \"0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866\": container with ID starting with 0a16dd5f42750f56a716a120d616a0c6259e39cdaaa2400f9d64a90ed294c866 not found: ID does not exist" Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.379699 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\") pod \"538963a2-cf66-4269-a2bc-30a8efb9b205\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.379839 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\") pod \"538963a2-cf66-4269-a2bc-30a8efb9b205\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.379935 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\") pod \"538963a2-cf66-4269-a2bc-30a8efb9b205\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " Oct 30 17:52:22.395041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.380101 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\" (UniqueName: \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\") pod \"538963a2-cf66-4269-a2bc-30a8efb9b205\" (UID: \"538963a2-cf66-4269-a2bc-30a8efb9b205\") " Oct 30 17:52:22.403302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.398773 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca" pod "538963a2-cf66-4269-a2bc-30a8efb9b205" (UID: "538963a2-cf66-4269-a2bc-30a8efb9b205"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:22.407448 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.407399 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc" pod "538963a2-cf66-4269-a2bc-30a8efb9b205" (UID: "538963a2-cf66-4269-a2bc-30a8efb9b205"). InnerVolumeSpecName "kube-api-access-h69bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:22.451335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.448965 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls" pod "538963a2-cf66-4269-a2bc-30a8efb9b205" (UID: "538963a2-cf66-4269-a2bc-30a8efb9b205"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:22.454321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.453573 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs" pod "538963a2-cf66-4269-a2bc-30a8efb9b205" (UID: "538963a2-cf66-4269-a2bc-30a8efb9b205"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:22.483219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.482778 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/538963a2-cf66-4269-a2bc-30a8efb9b205-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:22.483219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.482813 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:22.483219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.482828 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69bc\" (UniqueName: \"kubernetes.io/projected/538963a2-cf66-4269-a2bc-30a8efb9b205-kube-api-access-h69bc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:22.483219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.482841 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/538963a2-cf66-4269-a2bc-30a8efb9b205-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:22.695445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.695412 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:52:22.698985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:22.698959 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-488c67-5fcb8f68cf-mwv7g"] Oct 30 17:52:24.375841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:24.375807 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538963a2-cf66-4269-a2bc-30a8efb9b205" path="/var/lib/kubelet/pods/538963a2-cf66-4269-a2bc-30a8efb9b205/volumes" Oct 30 17:52:24.535624 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:52:24.535595 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.619680 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620652 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" containerName="main" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620674 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" containerName="main" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620692 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620701 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620714 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620723 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e380bc6-eabc-4c06-8a34-ce1e5919c599" containerName="istio-proxy" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620785 4622 cpu_manager.go:414] "containerMap: removing container" podUID="538963a2-cf66-4269-a2bc-30a8efb9b205" containerName="maas-api" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620793 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538963a2-cf66-4269-a2bc-30a8efb9b205" containerName="maas-api" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620939 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="538963a2-cf66-4269-a2bc-30a8efb9b205" containerName="maas-api" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620957 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c2f1f7-a15c-4011-bb85-65cc90fc59c0" containerName="main" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620968 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7604ad68-9342-45d8-be4d-4af284543004" containerName="istio-proxy" Oct 30 17:52:28.626306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.620980 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a116870-e9f7-41d5-98cc-828575727fd8" containerName="istio-proxy" Oct 30 17:52:28.631708 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.631678 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.675459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.675408 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-c31c73-openshift-default-dockercfg-qnblk\"" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676401 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676471 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676507 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676550 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676688 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676791 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676838 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676879 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84tz\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.676932 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.678350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.677077 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 17:52:28.778101 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778065 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778172 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778217 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778251 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84tz\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778336 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778392 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778448 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778482 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778542 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778565 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778771 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778833 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.778928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.778870 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.779179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.779159 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.780744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.780720 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.780838 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.780817 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.802344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.802319 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84tz\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.802550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.802531 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\") pod \"e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:28.986157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:28.986086 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:29.306675 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:52:29.306648 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4ca4c0_82e5_4322_96ee_81a62566f428.slice/crio-20cd508da595c6bd39ff2f2d5b07f70241dc6931587c991866f35fc47bd3b6e2 WatchSource:0}: Error finding container 20cd508da595c6bd39ff2f2d5b07f70241dc6931587c991866f35fc47bd3b6e2: Status 404 returned error can't find the container with id 20cd508da595c6bd39ff2f2d5b07f70241dc6931587c991866f35fc47bd3b6e2 Oct 30 17:52:29.308977 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.308948 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 17:52:29.309698 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.309667 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:29.309769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.309738 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:29.309822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.309783 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:29.395529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.395500 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" event={"ID":"be4ca4c0-82e5-4322-96ee-81a62566f428","Type":"ContainerStarted","Data":"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2"} Oct 30 17:52:29.395636 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.395555 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" event={"ID":"be4ca4c0-82e5-4322-96ee-81a62566f428","Type":"ContainerStarted","Data":"20cd508da595c6bd39ff2f2d5b07f70241dc6931587c991866f35fc47bd3b6e2"} Oct 30 17:52:29.444486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.444435 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podStartSLOduration=1.444422364 podStartE2EDuration="1.444422364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:29.440080144 +0000 UTC m=+2377.217552110" watchObservedRunningTime="2026-10-30 17:52:29.444422364 +0000 UTC m=+2377.221894322" Oct 30 17:52:29.986292 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:29.986240 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:30.295327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:30.293890 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:52:30.295327 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:30.294114 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" containerID="cri-o://f5b322cb1cc27c0cf962d79cc5c8394a06ba00d06fd64e03b0469ca4a2684b2b" gracePeriod=30 Oct 30 17:52:30.988428 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:30.988388 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:52:30.988932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:30.988444 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:52:31.993302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:31.991668 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:52:31.993302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:31.991738 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:52:32.986544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:32.986511 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:52:32.986717 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:32.986578 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:52:33.986378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:33.986345 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:52:33.986788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:33.986398 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:52:34.986438 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:34.986407 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:52:34.986827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:34.986455 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 17:52:35.445671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.445632 4622 generic.go:356] "Generic (PLEG): container finished" podID="d8df3750-cc02-42da-8e32-694298a39b89" containerID="f5b322cb1cc27c0cf962d79cc5c8394a06ba00d06fd64e03b0469ca4a2684b2b" exitCode=0 Oct 30 17:52:35.445856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.445702 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerDied","Data":"f5b322cb1cc27c0cf962d79cc5c8394a06ba00d06fd64e03b0469ca4a2684b2b"} Oct 30 17:52:35.445856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.445758 4622 scope.go:122] "RemoveContainer" containerID="5694eb7dbd5af0866d7d2134c3892e7c2ff006932573b59a31853a8a7f532dd0" Oct 30 17:52:35.579133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.579078 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:52:35.712835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.712795 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.712991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.712889 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.712991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.712935 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.712991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.712973 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713170 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713198 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713255 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713241 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713350 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713394 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\") pod \"d8df3750-cc02-42da-8e32-694298a39b89\" (UID: \"d8df3750-cc02-42da-8e32-694298a39b89\") " Oct 30 17:52:35.713521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713421 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:35.713521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713444 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:35.714230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713656 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:35.714230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.713988 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.714230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.714014 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.714230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.714026 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.714230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.714028 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:35.714918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.714888 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:35.715630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.715605 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:35.715854 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.715824 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "kube-api-access-cmm7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:35.715971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.715892 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:52:35.716735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.716710 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token" pod "d8df3750-cc02-42da-8e32-694298a39b89" (UID: "d8df3750-cc02-42da-8e32-694298a39b89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:35.816147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816108 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8df3750-cc02-42da-8e32-694298a39b89-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.816147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816147 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.816390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816164 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmm7b\" (UniqueName: \"kubernetes.io/projected/d8df3750-cc02-42da-8e32-694298a39b89-kube-api-access-cmm7b\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.816390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816189 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.816390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816203 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8df3750-cc02-42da-8e32-694298a39b89-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:35.816390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:35.816217 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8df3750-cc02-42da-8e32-694298a39b89-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:36.070965 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.070933 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:36.071432 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.071311 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:36.072050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.072026 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 17:52:36.453824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.453789 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" Oct 30 17:52:36.453979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.453817 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p" event={"ID":"d8df3750-cc02-42da-8e32-694298a39b89","Type":"ContainerDied","Data":"4e96848163776d858943de9ea440de8039d84590065f2e73abf7288f1bcdd699"} Oct 30 17:52:36.453979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.453872 4622 scope.go:122] "RemoveContainer" containerID="f5b322cb1cc27c0cf962d79cc5c8394a06ba00d06fd64e03b0469ca4a2684b2b" Oct 30 17:52:36.508609 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.508584 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:52:36.521423 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:36.521402 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-488c67-openshift-default-66fc49d7f5-9wv7p"] Oct 30 17:52:37.291131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.291095 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:52:37.291601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.291389 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" podUID="2c829eb5-e40d-493a-bca0-4f8232debac7" containerName="maas-api" containerID="cri-o://ee203e857a6e870ab3a3f159f6845e6ebe083698498d2ee39b20726da6053891" gracePeriod=30 Oct 30 17:52:37.464317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.464250 4622 generic.go:356] "Generic (PLEG): container finished" podID="2c829eb5-e40d-493a-bca0-4f8232debac7" containerID="ee203e857a6e870ab3a3f159f6845e6ebe083698498d2ee39b20726da6053891" exitCode=0 Oct 30 17:52:37.464498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.464290 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" event={"ID":"2c829eb5-e40d-493a-bca0-4f8232debac7","Type":"ContainerDied","Data":"ee203e857a6e870ab3a3f159f6845e6ebe083698498d2ee39b20726da6053891"} Oct 30 17:52:37.799309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.774892 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:52:37.850341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.848100 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\") pod \"2c829eb5-e40d-493a-bca0-4f8232debac7\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " Oct 30 17:52:37.850341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.848503 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca" pod "2c829eb5-e40d-493a-bca0-4f8232debac7" (UID: "2c829eb5-e40d-493a-bca0-4f8232debac7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:52:37.850341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.848720 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\") pod \"2c829eb5-e40d-493a-bca0-4f8232debac7\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " Oct 30 17:52:37.850341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.849338 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\") pod \"2c829eb5-e40d-493a-bca0-4f8232debac7\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " Oct 30 17:52:37.850341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.849687 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\" (UniqueName: \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\") pod \"2c829eb5-e40d-493a-bca0-4f8232debac7\" (UID: \"2c829eb5-e40d-493a-bca0-4f8232debac7\") " Oct 30 17:52:37.850735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.850658 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c829eb5-e40d-493a-bca0-4f8232debac7-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:37.862296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.861146 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs" pod "2c829eb5-e40d-493a-bca0-4f8232debac7" (UID: "2c829eb5-e40d-493a-bca0-4f8232debac7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:37.862296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.861280 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls" pod "2c829eb5-e40d-493a-bca0-4f8232debac7" (UID: "2c829eb5-e40d-493a-bca0-4f8232debac7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:37.862296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.861841 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2" pod "2c829eb5-e40d-493a-bca0-4f8232debac7" (UID: "2c829eb5-e40d-493a-bca0-4f8232debac7"). InnerVolumeSpecName "kube-api-access-fhlw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:37.951813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.951777 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:37.951813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.951811 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c829eb5-e40d-493a-bca0-4f8232debac7-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:37.951813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:37.951820 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlw2\" (UniqueName: \"kubernetes.io/projected/2c829eb5-e40d-493a-bca0-4f8232debac7-kube-api-access-fhlw2\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:52:38.377666 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.377629 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8df3750-cc02-42da-8e32-694298a39b89" path="/var/lib/kubelet/pods/d8df3750-cc02-42da-8e32-694298a39b89/volumes" Oct 30 17:52:38.473889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.473855 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" event={"ID":"2c829eb5-e40d-493a-bca0-4f8232debac7","Type":"ContainerDied","Data":"58bf910bb9516ed675e16a657252a0bb662580982d05235dc1565cfc1b3b17a1"} Oct 30 17:52:38.474040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.473897 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4" Oct 30 17:52:38.474040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.473916 4622 scope.go:122] "RemoveContainer" containerID="ee203e857a6e870ab3a3f159f6845e6ebe083698498d2ee39b20726da6053891" Oct 30 17:52:38.526763 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.526741 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:52:38.538827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:38.538803 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b1c345-7f4dc9967d-mtll4"] Oct 30 17:52:40.375825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:40.375792 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c829eb5-e40d-493a-bca0-4f8232debac7" path="/var/lib/kubelet/pods/2c829eb5-e40d-493a-bca0-4f8232debac7/volumes" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.643800 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644711 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644731 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644811 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2c829eb5-e40d-493a-bca0-4f8232debac7" containerName="maas-api" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644819 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c829eb5-e40d-493a-bca0-4f8232debac7" containerName="maas-api" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644955 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644970 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:46.647305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.644992 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c829eb5-e40d-493a-bca0-4f8232debac7" containerName="maas-api" Oct 30 17:52:46.650914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.650849 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.678028 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.676122 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 17:52:46.683287 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.681470 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-23c1e8-openshift-default-dockercfg-4mhhv\"" Oct 30 17:52:46.813981 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.813936 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.814244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.814222 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.814427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.814404 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.814644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.814624 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.824445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.824402 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.824758 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.824704 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmn7q\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.825036 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.825006 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.825337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.825312 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.825736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.825696 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.938607 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.938512 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.939665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.939643 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.940365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.940343 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.940892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.940872 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.941049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.941033 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.941180 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.941165 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.941339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.941310 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.942498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.942478 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.942663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.942632 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmn7q\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.942908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.941981 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.943029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.942365 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.943134 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.941794 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.947531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.947509 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.947661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.939567 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.979705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.960792 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.985175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.979256 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmn7q\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.985175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.973256 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:46.987670 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:46.987639 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\") pod \"e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:47.278423 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:47.278328 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:47.615305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:47.612857 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 17:52:47.615305 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:52:47.614127 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d32316_94a5_4484_b67a_387c737461e3.slice/crio-c75029408ce9d15847768a8491ae3c0d0ec5d4df1ac5bd51a91605f70280279a WatchSource:0}: Error finding container c75029408ce9d15847768a8491ae3c0d0ec5d4df1ac5bd51a91605f70280279a: Status 404 returned error can't find the container with id c75029408ce9d15847768a8491ae3c0d0ec5d4df1ac5bd51a91605f70280279a Oct 30 17:52:47.617752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:47.617719 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:47.617870 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:47.617792 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:47.617870 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:47.617824 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:52:48.577453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.577407 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerStarted","Data":"6d0f3cfe18082d56e5acb5f1885a0dbfcdc4bae1a1b50c20a0140f959cff011d"} Oct 30 17:52:48.577905 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.577478 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerStarted","Data":"c75029408ce9d15847768a8491ae3c0d0ec5d4df1ac5bd51a91605f70280279a"} Oct 30 17:52:48.646773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.646722 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podStartSLOduration=2.646710124 podStartE2EDuration="2.646710124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:48.633959336 +0000 UTC m=+2396.411431290" watchObservedRunningTime="2026-10-30 17:52:48.646710124 +0000 UTC m=+2396.424182143" Oct 30 17:52:48.881749 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.881667 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:52:48.882317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.882296 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:48.882317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.882315 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8df3750-cc02-42da-8e32-694298a39b89" containerName="istio-proxy" Oct 30 17:52:48.886459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.886435 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.903597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.903573 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ce0480\"/\"openshift-service-ca.crt\"" Oct 30 17:52:48.903597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.903587 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-ce0480\"/\"kube-root-ca.crt\"" Oct 30 17:52:48.903764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.903705 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ce0480\"/\"auth-test-model-ce0480-kserve-self-signed-certs\"" Oct 30 17:52:48.904560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.904538 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:52:48.912496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.912454 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-ce0480\"/\"default-dockercfg-pb4cj\"" Oct 30 17:52:48.985139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985101 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.985388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985191 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.985388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985236 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.985530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985432 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.985590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985554 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:48.985647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:48.985602 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086506 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086470 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086562 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086594 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086664 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086695 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.086778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.086735 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.087151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.087116 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.087324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.087170 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.087324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.087201 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.089373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.089339 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.089514 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.089422 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.119116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.119047 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\") pod \"auth-test-model-ce0480-kserve-588456cd57-f4nq8\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.210713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.206679 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:49.281850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.281329 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:49.293650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.293624 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:49.594100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.593921 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:49.596034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.596016 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:52:49.645331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:49.645301 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:52:49.646321 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:52:49.646257 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d967223_c9d6_46c5_ab6d_5b72fbd0a778.slice/crio-56d1d2a129487ebaf50cdce4f5400c08705d86086f0356544bac6931d1e30649 WatchSource:0}: Error finding container 56d1d2a129487ebaf50cdce4f5400c08705d86086f0356544bac6931d1e30649: Status 404 returned error can't find the container with id 56d1d2a129487ebaf50cdce4f5400c08705d86086f0356544bac6931d1e30649 Oct 30 17:52:50.601187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:50.601142 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" event={"ID":"2d967223-c9d6-46c5-ab6d-5b72fbd0a778","Type":"ContainerStarted","Data":"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e"} Oct 30 17:52:50.601637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:50.601196 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" event={"ID":"2d967223-c9d6-46c5-ab6d-5b72fbd0a778","Type":"ContainerStarted","Data":"56d1d2a129487ebaf50cdce4f5400c08705d86086f0356544bac6931d1e30649"} Oct 30 17:52:50.645754 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:50.645711 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" podStartSLOduration=2.645698329 podStartE2EDuration="2.645698329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:52:50.642925117 +0000 UTC m=+2398.420397073" watchObservedRunningTime="2026-10-30 17:52:50.645698329 +0000 UTC m=+2398.423170309" Oct 30 17:52:51.207290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:51.207227 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:54.534361 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:52:54.534328 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:52:59.207861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:59.207827 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:59.220909 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:59.220883 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:52:59.700594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:52:59.700564 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:53:15.043242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.043206 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 17:53:15.047544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.047521 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 17:53:15.047816 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.047800 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.065921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.065892 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-d53ec1-openshift-default-dockercfg-2q58v\"" Oct 30 17:53:15.133928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.133893 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.134194 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.134168 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.134440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.134420 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.134573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.134558 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.134716 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.134703 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.134974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.134958 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.135092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.135075 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.136041 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.136003 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.136353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.136319 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htsg2\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.237982 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238020 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238111 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238305 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238375 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238343 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238394 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238394 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238444 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238492 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htsg2\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238553 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238594 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.238780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238734 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.239012 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.238988 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.239113 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.239090 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.240620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.240598 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.240839 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.240823 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.262417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.262394 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.262714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.262692 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htsg2\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.426771 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.426691 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:15.756952 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:53:15.756920 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e53580_8fc3_452d_ad54_58b3ba82215a.slice/crio-9d867fa4190235b1fd040cb787f0c39141c8f2b33c2fce71944d4a9a3aa60bad WatchSource:0}: Error finding container 9d867fa4190235b1fd040cb787f0c39141c8f2b33c2fce71944d4a9a3aa60bad: Status 404 returned error can't find the container with id 9d867fa4190235b1fd040cb787f0c39141c8f2b33c2fce71944d4a9a3aa60bad Oct 30 17:53:15.757299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.757254 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 17:53:15.759612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.759583 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:53:15.759690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.759649 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:53:15.759690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.759678 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:53:15.812442 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:15.812404 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" event={"ID":"a7e53580-8fc3-452d-ad54-58b3ba82215a","Type":"ContainerStarted","Data":"9d867fa4190235b1fd040cb787f0c39141c8f2b33c2fce71944d4a9a3aa60bad"} Oct 30 17:53:16.820583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:16.820547 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" event={"ID":"a7e53580-8fc3-452d-ad54-58b3ba82215a","Type":"ContainerStarted","Data":"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76"} Oct 30 17:53:16.904450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:16.904396 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podStartSLOduration=2.904381217 podStartE2EDuration="2.904381217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:16.889134385 +0000 UTC m=+2424.666606340" watchObservedRunningTime="2026-10-30 17:53:16.904381217 +0000 UTC m=+2424.681853172" Oct 30 17:53:17.427331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:17.427291 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:17.428759 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:17.428735 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 17:53:17.428877 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:17.428782 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 17:53:18.427422 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:18.427387 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 17:53:18.427951 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:18.427449 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 17:53:19.427335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:19.427302 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 17:53:19.427513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:19.427351 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 17:53:20.293706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:20.293671 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:53:20.431460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:20.431406 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 17:53:20.432124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:20.432059 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 17:53:21.427192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:21.427145 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 17:53:21.427371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:21.427210 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 17:53:22.433048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:22.433021 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:22.433529 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:22.433512 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:22.434325 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:22.434307 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 17:53:24.534538 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:53:24.534505 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:53:32.724014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:32.723970 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:53:32.725471 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:32.725177 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" containerID="cri-o://0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030" gracePeriod=30 Oct 30 17:53:33.049874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:33.049826 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:53:35.327038 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.326999 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerName="main" containerID="cri-o://87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd" gracePeriod=45 Oct 30 17:53:35.606523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.606497 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:53:35.701792 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.701758 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702145 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702122 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\" (UniqueName: \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702169 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702296 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702466 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702346 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702466 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702449 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\") pod \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\" (UID: \"82a6ced5-347f-4dbb-92ed-6edb421a03c0\") " Oct 30 17:53:35.702850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702823 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:35.702980 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.702945 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:35.703079 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.703051 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:35.703316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.703283 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:35.705298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.705233 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:35.705444 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.705423 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "kube-api-access-l4j4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:35.708517 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.708487 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs" pod "82a6ced5-347f-4dbb-92ed-6edb421a03c0" (UID: "82a6ced5-347f-4dbb-92ed-6edb421a03c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:35.804630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.804595 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:35.804630 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.804627 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:35.804818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.804643 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4j4d\" (UniqueName: \"kubernetes.io/projected/82a6ced5-347f-4dbb-92ed-6edb421a03c0-kube-api-access-l4j4d\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:35.804818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.804657 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:35.804818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:35.804670 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a6ced5-347f-4dbb-92ed-6edb421a03c0-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:36.025590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.025513 4622 generic.go:356] "Generic (PLEG): container finished" podID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerID="87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd" exitCode=0 Oct 30 17:53:36.025590 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.025556 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" Oct 30 17:53:36.025776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.025596 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" event={"ID":"82a6ced5-347f-4dbb-92ed-6edb421a03c0","Type":"ContainerDied","Data":"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd"} Oct 30 17:53:36.025776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.025638 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" event={"ID":"82a6ced5-347f-4dbb-92ed-6edb421a03c0","Type":"ContainerDied","Data":"72a917be5d35704826d51cf5947759c17b1fe03aaa176dc0739f027591e58898"} Oct 30 17:53:36.025776 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.025656 4622 scope.go:122] "RemoveContainer" containerID="87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd" Oct 30 17:53:36.036581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.036558 4622 scope.go:122] "RemoveContainer" containerID="87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd" Oct 30 17:53:36.036829 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:53:36.036809 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd\": container with ID starting with 87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd not found: ID does not exist" containerID="87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd" Oct 30 17:53:36.036903 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.036841 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd"} err="failed to get container status \"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd\": rpc error: code = NotFound desc = could not find container \"87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd\": container with ID starting with 87bd64d5e96c52056694838093efb6c65e2a9ce9a2484accab664abeb485d5fd not found: ID does not exist" Oct 30 17:53:36.079155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.079133 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:53:36.082597 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.082576 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t"] Oct 30 17:53:36.374671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.374640 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" path="/var/lib/kubelet/pods/82a6ced5-347f-4dbb-92ed-6edb421a03c0/volumes" Oct 30 17:53:36.588288 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.581357 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-043253/test-model-043253-kserve-5955984d9c-g9t4t" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": context deadline exceeded" Oct 30 17:53:36.847521 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.847471 4622 patch_prober.go:29] interesting pod/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:53:36.847738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:36.847550 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:53:38.015004 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.014979 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:53:38.058196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.058164 4622 generic.go:356] "Generic (PLEG): container finished" podID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerID="0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030" exitCode=0 Oct 30 17:53:38.058407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.058218 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" Oct 30 17:53:38.058407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.058250 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" event={"ID":"d1cc1f6d-f714-4171-ab66-9d97a342da42","Type":"ContainerDied","Data":"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030"} Oct 30 17:53:38.058407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.058349 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq" event={"ID":"d1cc1f6d-f714-4171-ab66-9d97a342da42","Type":"ContainerDied","Data":"fff6bda0efafde2d8c856b2cf987354642c18e9273c848c7e1ae74a58639ef25"} Oct 30 17:53:38.058407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.058383 4622 scope.go:122] "RemoveContainer" containerID="0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030" Oct 30 17:53:38.072766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.072286 4622 scope.go:122] "RemoveContainer" containerID="0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030" Oct 30 17:53:38.073818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.073672 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.073818 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:53:38.073673 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030\": container with ID starting with 0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030 not found: ID does not exist" containerID="0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030" Oct 30 17:53:38.073818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.073729 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030"} err="failed to get container status \"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030\": rpc error: code = NotFound desc = could not find container \"0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030\": container with ID starting with 0912a5c0618191a64064a3a942aefc212458196c821b9ea46362fb25f6f04030 not found: ID does not exist" Oct 30 17:53:38.076224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076002 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.076224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076065 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.076224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076125 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.076224 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076163 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.076645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076533 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.077152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076734 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.077152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076814 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.077152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076904 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.077152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.076934 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\") pod \"d1cc1f6d-f714-4171-ab66-9d97a342da42\" (UID: \"d1cc1f6d-f714-4171-ab66-9d97a342da42\") " Oct 30 17:53:38.078187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.077993 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.078187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.078104 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:38.078681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.078409 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.079477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.079406 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.081650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.081624 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "kube-api-access-gqtt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.082908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.082831 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.085135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.085085 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:53:38.085338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.085317 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:38.086751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.086729 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token" pod "d1cc1f6d-f714-4171-ab66-9d97a342da42" (UID: "d1cc1f6d-f714-4171-ab66-9d97a342da42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:38.179140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179108 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179141 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179157 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqtt4\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-kube-api-access-gqtt4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179166 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1cc1f6d-f714-4171-ab66-9d97a342da42-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179174 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179182 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1cc1f6d-f714-4171-ab66-9d97a342da42-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179189 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.179352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.179197 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1cc1f6d-f714-4171-ab66-9d97a342da42-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:38.416495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.416466 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:53:38.428872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:38.428849 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-303ba5-openshift-default-65b45b8ddf-6d2hq"] Oct 30 17:53:40.377148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:40.377117 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" path="/var/lib/kubelet/pods/d1cc1f6d-f714-4171-ab66-9d97a342da42/volumes" Oct 30 17:53:48.080397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.080354 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" podUID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" containerName="main" containerID="cri-o://9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d" gracePeriod=45 Oct 30 17:53:48.401876 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.401841 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:53:48.402647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402622 4622 cpu_manager.go:414] "containerMap: removing container" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerName="main" Oct 30 17:53:48.402647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402646 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerName="main" Oct 30 17:53:48.402818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402676 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" Oct 30 17:53:48.402818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402684 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" Oct 30 17:53:48.402921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402900 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a6ced5-347f-4dbb-92ed-6edb421a03c0" containerName="main" Oct 30 17:53:48.402921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.402914 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1cc1f6d-f714-4171-ab66-9d97a342da42" containerName="istio-proxy" Oct 30 17:53:48.406939 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.406917 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.424304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.418717 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:53:48.424304 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.421937 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-01b925\"/\"kube-root-ca.crt\"" Oct 30 17:53:48.427390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.427292 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-01b925\"/\"openshift-service-ca.crt\"" Oct 30 17:53:48.431307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431255 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.431515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431494 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.431573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431549 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqjq\" (UniqueName: \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.431829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431649 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.431829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431753 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.431963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.431849 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.434513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.434489 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-01b925\"/\"default-dockercfg-5m8rd\"" Oct 30 17:53:48.434611 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.434598 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-01b925\"/\"auth-test-model-01b925-kserve-self-signed-certs\"" Oct 30 17:53:48.440177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.438525 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:53:48.532490 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.532451 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.532684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.532503 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\" (UniqueName: \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.532684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.532573 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.532843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.532820 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:48.533021 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.532999 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.533235 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533214 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.533366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533321 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\") pod \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\" (UID: \"4c651c3b-34d0-4d27-992d-86a10f88e5e3\") " Oct 30 17:53:48.533601 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533580 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.533688 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqjq\" (UniqueName: \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.533775 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533761 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.533846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533803 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.533907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533867 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.533962 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.533907 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.534131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.534118 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.534544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.534523 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.535016 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.534976 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.535376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.535355 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.535556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.535528 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:48.535843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.535819 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:48.537144 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.537118 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:48.537350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.537320 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:53:48.538650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.538612 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.539060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.539037 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29" pod "4c651c3b-34d0-4d27-992d-86a10f88e5e3" (UID: "4c651c3b-34d0-4d27-992d-86a10f88e5e3"). InnerVolumeSpecName "kube-api-access-ghz29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:48.540592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.540570 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.570869 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.570835 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqjq\" (UniqueName: \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\") pod \"auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:48.635880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.635843 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.635880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.635876 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.635880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.635889 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.636185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.635904 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c651c3b-34d0-4d27-992d-86a10f88e5e3-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.636185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.635920 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghz29\" (UniqueName: \"kubernetes.io/projected/4c651c3b-34d0-4d27-992d-86a10f88e5e3-kube-api-access-ghz29\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:53:48.751389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:48.749655 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:49.112789 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:53:49.112757 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8f5a05_f021_4fcb_92b0_55d66e41334b.slice/crio-3917ac0883f4e7d591148e6e4e634e46ba219487d49e20b5c1b0ccbe673c807b WatchSource:0}: Error finding container 3917ac0883f4e7d591148e6e4e634e46ba219487d49e20b5c1b0ccbe673c807b: Status 404 returned error can't find the container with id 3917ac0883f4e7d591148e6e4e634e46ba219487d49e20b5c1b0ccbe673c807b Oct 30 17:53:49.113990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.113961 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:53:49.159864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.159832 4622 generic.go:356] "Generic (PLEG): container finished" podID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" containerID="9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d" exitCode=0 Oct 30 17:53:49.159999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.159886 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" Oct 30 17:53:49.159999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.159913 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" event={"ID":"4c651c3b-34d0-4d27-992d-86a10f88e5e3","Type":"ContainerDied","Data":"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d"} Oct 30 17:53:49.159999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.159964 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw" event={"ID":"4c651c3b-34d0-4d27-992d-86a10f88e5e3","Type":"ContainerDied","Data":"13bcae6342406b53db873bdf52e9529e6272c01bbb3565a7d821b2bbe12a4bba"} Oct 30 17:53:49.159999 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.159988 4622 scope.go:122] "RemoveContainer" containerID="9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d" Oct 30 17:53:49.161873 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.161848 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" event={"ID":"bd8f5a05-f021-4fcb-92b0-55d66e41334b","Type":"ContainerStarted","Data":"3917ac0883f4e7d591148e6e4e634e46ba219487d49e20b5c1b0ccbe673c807b"} Oct 30 17:53:49.171837 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.171819 4622 scope.go:122] "RemoveContainer" containerID="9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d" Oct 30 17:53:49.172117 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:53:49.172089 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d\": container with ID starting with 9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d not found: ID does not exist" containerID="9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d" Oct 30 17:53:49.172172 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.172127 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d"} err="failed to get container status \"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d\": rpc error: code = NotFound desc = could not find container \"9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d\": container with ID starting with 9648e131e2cb2db2cbf615eba24a455d93ec6c17db06ca33f3a1305141f7ef6d not found: ID does not exist" Oct 30 17:53:49.217346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.217317 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:53:49.226839 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:49.226811 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bcdac0/test-model-bcdac0-kserve-7cf6cfd4b9-jfcdw"] Oct 30 17:53:50.172564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:50.172529 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" event={"ID":"bd8f5a05-f021-4fcb-92b0-55d66e41334b","Type":"ContainerStarted","Data":"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1"} Oct 30 17:53:50.214902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:50.214857 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" podStartSLOduration=2.214841073 podStartE2EDuration="2.214841073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:50.213570333 +0000 UTC m=+2457.991042299" watchObservedRunningTime="2026-10-30 17:53:50.214841073 +0000 UTC m=+2457.992313030" Oct 30 17:53:50.376049 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:50.376004 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" path="/var/lib/kubelet/pods/4c651c3b-34d0-4d27-992d-86a10f88e5e3/volumes" Oct 30 17:53:50.750974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:50.750878 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:54.534127 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:53:54.534098 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:53:57.063301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:57.063253 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:53:57.063720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:57.063539 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" containerID="cri-o://7c87b1df0f61ac2e026f62dda7d6f53fcf7325d29a3753a90634888eec48e8c1" gracePeriod=30 Oct 30 17:53:57.260986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:57.260950 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:53:57.261173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:57.261135 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" containerID="cri-o://4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97" gracePeriod=30 Oct 30 17:53:58.750775 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:58.750740 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:58.763682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:58.763657 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:53:59.249248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:53:59.249218 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:54:00.402249 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:00.402221 4622 patch_prober.go:29] interesting pod/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:54:00.402664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:00.402304 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:54:01.067162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:01.067128 4622 patch_prober.go:29] interesting pod/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:54:01.067373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:01.067187 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:54:02.262766 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.262733 4622 generic.go:356] "Generic (PLEG): container finished" podID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerID="7c87b1df0f61ac2e026f62dda7d6f53fcf7325d29a3753a90634888eec48e8c1" exitCode=0 Oct 30 17:54:02.263208 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.262813 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" event={"ID":"887b5174-c727-4d7f-bf67-bedbd15eb7cd","Type":"ContainerDied","Data":"7c87b1df0f61ac2e026f62dda7d6f53fcf7325d29a3753a90634888eec48e8c1"} Oct 30 17:54:02.396518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.396464 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:54:02.527463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.527438 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:54:02.529344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529322 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529369 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529450 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529402 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529528 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529628 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529605 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.529684 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529662 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529730 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529780 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529813 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529782 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.529927 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529823 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:02.529986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529952 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.529986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529962 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.530066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529984 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.530066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.529992 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\") pod \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\" (UID: \"887b5174-c727-4d7f-bf67-bedbd15eb7cd\") " Oct 30 17:54:02.530726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.530700 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.530847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.530730 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.530847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.530744 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.530847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.530756 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.530847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.530771 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.531809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.531784 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:02.531893 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.531807 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.532299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.532274 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:02.532299 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.532297 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j" pod "887b5174-c727-4d7f-bf67-bedbd15eb7cd" (UID: "887b5174-c727-4d7f-bf67-bedbd15eb7cd"). InnerVolumeSpecName "kube-api-access-cfm8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:02.631242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631206 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631325 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631412 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631445 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631520 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631596 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631632 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631642 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.631810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631663 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631810 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631782 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.631916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631848 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\") pod \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\" (UID: \"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0\") " Oct 30 17:54:02.631916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.631869 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.632078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632058 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.632185 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632120 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:02.632467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632444 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632469 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632484 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632499 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfm8j\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-kube-api-access-cfm8j\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632514 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632527 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/887b5174-c727-4d7f-bf67-bedbd15eb7cd-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632540 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632553 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.632593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.632566 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.633702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.633679 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:02.633971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.633946 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:02.634238 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.634215 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:02.634360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.634324 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw" pod "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" (UID: "6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0"). InnerVolumeSpecName "kube-api-access-4f4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:02.740627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.740525 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.740627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.740567 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.740627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.740584 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.740627 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.740597 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4sw\" (UniqueName: \"kubernetes.io/projected/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0-kube-api-access-4f4sw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:02.901296 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.901242 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:54:02.902151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902121 4622 cpu_manager.go:414] "containerMap: removing container" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" Oct 30 17:54:02.902151 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902150 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" Oct 30 17:54:02.902345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902170 4622 cpu_manager.go:414] "containerMap: removing container" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" Oct 30 17:54:02.902345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902179 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" Oct 30 17:54:02.902345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902207 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" containerName="main" Oct 30 17:54:02.902345 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902215 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" containerName="main" Oct 30 17:54:02.902472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902425 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" containerName="istio-proxy" Oct 30 17:54:02.902472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902443 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerName="istio-proxy" Oct 30 17:54:02.902472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.902455 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c651c3b-34d0-4d27-992d-86a10f88e5e3" containerName="main" Oct 30 17:54:02.907908 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.907888 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:02.923220 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.923179 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-8d1062\"/\"default-dockercfg-h65cd\"" Oct 30 17:54:02.923335 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.923298 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-8d1062\"/\"auth-test-model-8d1062-kserve-self-signed-certs\"" Oct 30 17:54:02.925958 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.925938 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:54:02.932565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.932547 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-8d1062\"/\"openshift-service-ca.crt\"" Oct 30 17:54:02.932669 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:02.932587 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-8d1062\"/\"kube-root-ca.crt\"" Oct 30 17:54:03.045350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045313 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.045543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045387 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.045543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045515 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.045720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045577 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.045720 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045693 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.045847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.045828 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4t6\" (UniqueName: \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.151240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.151191 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.151561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.151539 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.151728 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.151712 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.151856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.151843 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.151985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.151972 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.152175 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.152159 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4t6\" (UniqueName: \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.153361 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.153337 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.153757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.153738 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.159562 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.159539 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.159940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.159920 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.172281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.170197 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.179718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.179688 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4t6\" (UniqueName: \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\") pod \"auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.227029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.224820 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:03.281861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.281825 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" event={"ID":"887b5174-c727-4d7f-bf67-bedbd15eb7cd","Type":"ContainerDied","Data":"2dbab7e0698c391a767dd9bafb9e1f0c978614b157439f9623f9836a0ebc863c"} Oct 30 17:54:03.283032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.283010 4622 scope.go:122] "RemoveContainer" containerID="7c87b1df0f61ac2e026f62dda7d6f53fcf7325d29a3753a90634888eec48e8c1" Oct 30 17:54:03.283289 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.282467 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz" Oct 30 17:54:03.289104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.285232 4622 generic.go:356] "Generic (PLEG): container finished" podID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" containerID="4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97" exitCode=0 Oct 30 17:54:03.289104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.285295 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" event={"ID":"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0","Type":"ContainerDied","Data":"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97"} Oct 30 17:54:03.289104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.285334 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" event={"ID":"6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0","Type":"ContainerDied","Data":"de0024a3eb75a9d1c42651f5fdf0dbc7078cb825be0ec5f8452b538531ff6e10"} Oct 30 17:54:03.289104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.285405 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq" Oct 30 17:54:03.306149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.306112 4622 scope.go:122] "RemoveContainer" containerID="4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97" Oct 30 17:54:03.322697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.322672 4622 scope.go:122] "RemoveContainer" containerID="4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97" Oct 30 17:54:03.323341 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:03.323306 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97\": container with ID starting with 4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97 not found: ID does not exist" containerID="4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97" Oct 30 17:54:03.323433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.323358 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97"} err="failed to get container status \"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97\": rpc error: code = NotFound desc = could not find container \"4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97\": container with ID starting with 4ea54f2815ffdb367355c5185ee0e6c34b43ad05ef93b77cf882869fcd8b9e97 not found: ID does not exist" Oct 30 17:54:03.368391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.366910 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:54:03.370800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.370669 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0e7403-openshift-default-69b4d87b99-nt2hq"] Oct 30 17:54:03.403197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.403047 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:54:03.407010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.406983 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-db98b0-openshift-default-9f957cc9c-wnmdz"] Oct 30 17:54:03.690568 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:03.641476 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:54:04.349283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:04.342049 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" event={"ID":"e6338952-93f9-4634-b015-3ee549712d4d","Type":"ContainerStarted","Data":"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad"} Oct 30 17:54:04.349283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:04.342117 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" event={"ID":"e6338952-93f9-4634-b015-3ee549712d4d","Type":"ContainerStarted","Data":"a53fa13d6cd1bf8f56f2415774874c70bd5ca22550858b1ea3de08e775de1614"} Oct 30 17:54:04.439337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:04.439274 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" podStartSLOduration=2.439239607 podStartE2EDuration="2.439239607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:04.386856807 +0000 UTC m=+2472.164328763" watchObservedRunningTime="2026-10-30 17:54:04.439239607 +0000 UTC m=+2472.216711563" Oct 30 17:54:04.474351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:04.468761 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0" path="/var/lib/kubelet/pods/6d0b4b9c-19ad-4691-a2ff-9b2b315b15d0/volumes" Oct 30 17:54:04.474351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:04.473064 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887b5174-c727-4d7f-bf67-bedbd15eb7cd" path="/var/lib/kubelet/pods/887b5174-c727-4d7f-bf67-bedbd15eb7cd/volumes" Oct 30 17:54:05.225567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:05.225535 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:13.225434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:13.225398 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:13.238610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:13.238582 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:13.440043 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:13.440009 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:54:14.437240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.437208 4622 generic.go:356] "Generic (PLEG): container finished" podID="94d32316-94a5-4484-b67a-387c737461e3" containerID="6d0f3cfe18082d56e5acb5f1885a0dbfcdc4bae1a1b50c20a0140f959cff011d" exitCode=0 Oct 30 17:54:14.437647 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.437315 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerDied","Data":"6d0f3cfe18082d56e5acb5f1885a0dbfcdc4bae1a1b50c20a0140f959cff011d"} Oct 30 17:54:14.437899 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.437876 4622 scope.go:122] "RemoveContainer" containerID="6d0f3cfe18082d56e5acb5f1885a0dbfcdc4bae1a1b50c20a0140f959cff011d" Oct 30 17:54:14.439458 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.439428 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:54:14.439566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.439511 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:54:14.439566 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:14.439561 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:54:15.461285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:15.457225 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerStarted","Data":"7210d8d3498d1f4059584bc10cdef6f87cc96ac0ed16cabd1842935bf3d71bb1"} Oct 30 17:54:16.278560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:16.278528 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:54:16.279904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:16.279876 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 17:54:16.280047 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:16.279941 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 17:54:17.279679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.279650 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 17:54:17.280082 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.279719 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 17:54:17.683108 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.683072 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk"] Oct 30 17:54:17.687310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.687293 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.706724 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.706659 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk"] Oct 30 17:54:17.714504 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.714477 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:54:17.714867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.714846 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:54:17.779843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.779801 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.780014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.779876 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9gz\" (UniqueName: \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.780014 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.779989 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.891349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.888301 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.891349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.888585 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.891349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.888633 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9gz\" (UniqueName: \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.891349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.891150 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.895029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.895004 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:17.912139 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:17.912109 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9gz\" (UniqueName: \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\") pod \"maas-api-revoke-sub-65850e4d-w2rhk\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:18.003832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:18.003763 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:18.279532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:18.279432 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 17:54:18.279532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:18.279501 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 17:54:18.327552 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:54:18.327520 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f71e34a_5d72_4cfd_be6a_4b28ef395c4e.slice/crio-39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de WatchSource:0}: Error finding container 39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de: Status 404 returned error can't find the container with id 39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de Oct 30 17:54:18.327885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:18.327618 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk"] Oct 30 17:54:18.488460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:18.488421 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" event={"ID":"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e","Type":"ContainerStarted","Data":"39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de"} Oct 30 17:54:19.286372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:19.279813 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 17:54:19.286372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:19.281418 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 17:54:19.558337 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:19.540687 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" event={"ID":"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e","Type":"ContainerStarted","Data":"35510214c2c44c77f2d105389f8cd6948643bfb45af4156264d0e069f1de39c5"} Oct 30 17:54:19.595246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:19.594242 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:54:20.279984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:20.279937 4622 patch_prober.go:29] interesting pod/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body= Oct 30 17:54:20.280172 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:20.280005 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" Oct 30 17:54:20.572034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:20.572000 4622 generic.go:356] "Generic (PLEG): container finished" podID="2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" containerID="35510214c2c44c77f2d105389f8cd6948643bfb45af4156264d0e069f1de39c5" exitCode=0 Oct 30 17:54:20.572511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:20.572064 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" event={"ID":"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e","Type":"ContainerDied","Data":"35510214c2c44c77f2d105389f8cd6948643bfb45af4156264d0e069f1de39c5"} Oct 30 17:54:21.283528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.283498 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:54:21.284409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.284392 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 17:54:21.779360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.779332 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:21.816859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.816813 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:54:21.817092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.817063 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" containerID="cri-o://74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7" gracePeriod=30 Oct 30 17:54:21.831533 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.831497 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:54:21.831760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.831735 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" containerID="cri-o://4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656" gracePeriod=30 Oct 30 17:54:21.881485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.881223 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\") pod \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " Oct 30 17:54:21.881485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.881323 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\") pod \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " Oct 30 17:54:21.881485 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.881361 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\" (UniqueName: \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\") pod \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\" (UID: \"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e\") " Oct 30 17:54:21.885034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.882047 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca" pod "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" (UID: "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:21.885034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.883706 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:54:21.885034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.883994 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" containerID="cri-o://3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b" gracePeriod=30 Oct 30 17:54:21.887025 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.886806 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz" pod "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" (UID: "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e"). InnerVolumeSpecName "kube-api-access-bb9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:21.889888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.889847 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca" pod "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" (UID: "2f71e34a-5d72-4cfd-be6a-4b28ef395c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:21.983413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.983377 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:21.983413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.983415 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:21.983623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:21.983430 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9gz\" (UniqueName: \"kubernetes.io/projected/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e-kube-api-access-bb9gz\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:22.346244 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.346212 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc"] Oct 30 17:54:22.346844 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.346829 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" containerName="revoke-keys" Oct 30 17:54:22.346891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.346846 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" containerName="revoke-keys" Oct 30 17:54:22.346971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.346962 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" containerName="revoke-keys" Oct 30 17:54:22.350620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.350605 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.376380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.376352 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc"] Oct 30 17:54:22.391425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.391396 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxfl\" (UniqueName: \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.391550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.391459 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.391625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.391599 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.492626 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.492594 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.492800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.492787 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxfl\" (UniqueName: \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.492847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.492833 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.493473 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.493445 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.494865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.494838 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.516547 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.516526 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxfl\" (UniqueName: \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\") pod \"maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.592818 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.592793 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" Oct 30 17:54:22.592926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.592837 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk" event={"ID":"2f71e34a-5d72-4cfd-be6a-4b28ef395c4e","Type":"ContainerDied","Data":"39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de"} Oct 30 17:54:22.592926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.592879 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ca51b59f3bc82cfd1b763f7cb9f1f7d028dcc31f6cfa04be995d4f405948de" Oct 30 17:54:22.665350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.665283 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:22.968904 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:54:22.968874 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3db0f0f_479b_4e62_aed1_fca892affc27.slice/crio-e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190 WatchSource:0}: Error finding container e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190: Status 404 returned error can't find the container with id e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190 Oct 30 17:54:22.969378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:22.969064 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc"] Oct 30 17:54:23.601209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:23.601152 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" event={"ID":"b3db0f0f-479b-4e62-aed1-fca892affc27","Type":"ContainerStarted","Data":"6696d97136b6aeecf5033b07ba4a0588aac384f8b82888cd30d083b060dd5754"} Oct 30 17:54:23.601439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:23.601218 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" event={"ID":"b3db0f0f-479b-4e62-aed1-fca892affc27","Type":"ContainerStarted","Data":"e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190"} Oct 30 17:54:23.644677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:23.644633 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" podStartSLOduration=1.644619982 podStartE2EDuration="1.644619982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:23.640845183 +0000 UTC m=+2491.418317140" watchObservedRunningTime="2026-10-30 17:54:23.644619982 +0000 UTC m=+2491.422091937" Oct 30 17:54:24.534556 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:24.534525 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:54:24.613883 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:24.613854 4622 generic.go:356] "Generic (PLEG): container finished" podID="b3db0f0f-479b-4e62-aed1-fca892affc27" containerID="6696d97136b6aeecf5033b07ba4a0588aac384f8b82888cd30d083b060dd5754" exitCode=0 Oct 30 17:54:24.614039 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:24.613914 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" event={"ID":"b3db0f0f-479b-4e62-aed1-fca892affc27","Type":"ContainerDied","Data":"6696d97136b6aeecf5033b07ba4a0588aac384f8b82888cd30d083b060dd5754"} Oct 30 17:54:25.800514 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.800483 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:25.838301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.837237 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\") pod \"b3db0f0f-479b-4e62-aed1-fca892affc27\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " Oct 30 17:54:25.838301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.837377 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\") pod \"b3db0f0f-479b-4e62-aed1-fca892affc27\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " Oct 30 17:54:25.838301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.837478 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\" (UniqueName: \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\") pod \"b3db0f0f-479b-4e62-aed1-fca892affc27\" (UID: \"b3db0f0f-479b-4e62-aed1-fca892affc27\") " Oct 30 17:54:25.838657 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.838632 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca" pod "b3db0f0f-479b-4e62-aed1-fca892affc27" (UID: "b3db0f0f-479b-4e62-aed1-fca892affc27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:25.845475 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.845448 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca" pod "b3db0f0f-479b-4e62-aed1-fca892affc27" (UID: "b3db0f0f-479b-4e62-aed1-fca892affc27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:25.845640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.845477 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl" pod "b3db0f0f-479b-4e62-aed1-fca892affc27" (UID: "b3db0f0f-479b-4e62-aed1-fca892affc27"). InnerVolumeSpecName "kube-api-access-rjxfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:25.939492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.939397 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxfl\" (UniqueName: \"kubernetes.io/projected/b3db0f0f-479b-4e62-aed1-fca892affc27-kube-api-access-rjxfl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:25.939492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.939426 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:25.939492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:25.939435 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3db0f0f-479b-4e62-aed1-fca892affc27-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:26.628822 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:26.628792 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" Oct 30 17:54:26.628990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:26.628828 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc" event={"ID":"b3db0f0f-479b-4e62-aed1-fca892affc27","Type":"ContainerDied","Data":"e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190"} Oct 30 17:54:26.628990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:26.628881 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d63e7fa275e28387d8a97828e307fe90a609cc994b49c544fd78b3ee87b190" Oct 30 17:54:27.177252 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.177222 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:54:27.195565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.195545 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:54:27.223814 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.223793 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:54:27.258169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258138 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258212 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258244 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258308 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.258380 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258361 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258413 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258428 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258445 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258474 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258508 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258549 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258595 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258666 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258673 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258732 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258768 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258810 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258841 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258873 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258911 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.258986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258957 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.258995 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259061 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259120 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259154 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\") pod \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\" (UID: \"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259185 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259214 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\") pod \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\" (UID: \"e52fa5f6-44d5-4412-b1c5-c98fa850c47b\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259290 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.259472 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259393 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\") pod \"c9392dd6-86d2-498b-8a76-c84e06392bf8\" (UID: \"c9392dd6-86d2-498b-8a76-c84e06392bf8\") " Oct 30 17:54:27.259866 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259815 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.259967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.259938 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.260096 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260074 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.260188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260166 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.260374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260348 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260411 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260429 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260430 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260444 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260460 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.260487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.260476 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.261044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261019 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.261894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261312 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.261894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261489 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.261894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261534 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.261894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261593 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.262231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.261974 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.262231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.262172 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.263570 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.263544 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.264119 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.264069 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:27.264502 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.264476 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "kube-api-access-6vpc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.264722 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.264699 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.266498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.266391 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.266498 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.266475 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.267048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.266997 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "kube-api-access-jk8fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.267544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.267514 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:27.268057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.268028 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo" pod "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" (UID: "cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.268635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.268609 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.268948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.268915 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.269075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.269058 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9" pod "e52fa5f6-44d5-4412-b1c5-c98fa850c47b" (UID: "e52fa5f6-44d5-4412-b1c5-c98fa850c47b"). InnerVolumeSpecName "kube-api-access-mkcg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:27.269552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.269527 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo" pod "c9392dd6-86d2-498b-8a76-c84e06392bf8" (UID: "c9392dd6-86d2-498b-8a76-c84e06392bf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:27.361996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.361962 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8fc\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-kube-api-access-jk8fc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.361996 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.361996 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362010 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362023 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcg9\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-kube-api-access-mkcg9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362036 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362050 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362062 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362073 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362086 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9392dd6-86d2-498b-8a76-c84e06392bf8-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362098 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362110 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362121 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362135 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9392dd6-86d2-498b-8a76-c84e06392bf8-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362147 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vpc8\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-kube-api-access-6vpc8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362158 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362170 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362180 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362193 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362203 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362214 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e52fa5f6-44d5-4412-b1c5-c98fa850c47b-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.362284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.362225 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9392dd6-86d2-498b-8a76-c84e06392bf8-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:27.637344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.637307 4622 generic.go:356] "Generic (PLEG): container finished" podID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerID="74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7" exitCode=0 Oct 30 17:54:27.637538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.637364 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" Oct 30 17:54:27.637538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.637401 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" event={"ID":"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3","Type":"ContainerDied","Data":"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7"} Oct 30 17:54:27.637538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.637455 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj" event={"ID":"cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3","Type":"ContainerDied","Data":"611e43b74c2a2f052bca9d8cb0576359c57df6beb52f1f70c04617ee8e484aa3"} Oct 30 17:54:27.637538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.637481 4622 scope.go:122] "RemoveContainer" containerID="74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7" Oct 30 17:54:27.639064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.639038 4622 generic.go:356] "Generic (PLEG): container finished" podID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerID="3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b" exitCode=0 Oct 30 17:54:27.639319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.639071 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" event={"ID":"e52fa5f6-44d5-4412-b1c5-c98fa850c47b","Type":"ContainerDied","Data":"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b"} Oct 30 17:54:27.639449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.639113 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" Oct 30 17:54:27.639449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.639354 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv" event={"ID":"e52fa5f6-44d5-4412-b1c5-c98fa850c47b","Type":"ContainerDied","Data":"cb30be12ae5252963cc843a760bcfbe26923141b2c9bf32f9852264463cbc6a5"} Oct 30 17:54:27.646155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.646135 4622 generic.go:356] "Generic (PLEG): container finished" podID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerID="4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656" exitCode=0 Oct 30 17:54:27.646312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.646225 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" event={"ID":"c9392dd6-86d2-498b-8a76-c84e06392bf8","Type":"ContainerDied","Data":"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656"} Oct 30 17:54:27.646312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.646280 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" event={"ID":"c9392dd6-86d2-498b-8a76-c84e06392bf8","Type":"ContainerDied","Data":"21ac3f023043e8e823da7c03608b5f9a60c6ca7c7ebe564cb71944303734cf21"} Oct 30 17:54:27.646312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.646305 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk" Oct 30 17:54:27.649032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.649014 4622 scope.go:122] "RemoveContainer" containerID="74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7" Oct 30 17:54:27.649545 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:27.649321 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7\": container with ID starting with 74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7 not found: ID does not exist" containerID="74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7" Oct 30 17:54:27.649545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.649356 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7"} err="failed to get container status \"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7\": rpc error: code = NotFound desc = could not find container \"74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7\": container with ID starting with 74e5a2d89a603faeea706daf7b8fedc55d076493a5430f06bcf7d6691a03b0d7 not found: ID does not exist" Oct 30 17:54:27.649545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.649378 4622 scope.go:122] "RemoveContainer" containerID="3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b" Oct 30 17:54:27.659415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.659398 4622 scope.go:122] "RemoveContainer" containerID="3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b" Oct 30 17:54:27.659645 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:27.659630 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b\": container with ID starting with 3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b not found: ID does not exist" containerID="3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b" Oct 30 17:54:27.659691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.659652 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b"} err="failed to get container status \"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b\": rpc error: code = NotFound desc = could not find container \"3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b\": container with ID starting with 3eb00328e37c7623dfaf47da20c0c75b05eea6b9601838ee9785a62b6bdf9f9b not found: ID does not exist" Oct 30 17:54:27.659691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.659668 4622 scope.go:122] "RemoveContainer" containerID="4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656" Oct 30 17:54:27.668902 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.668881 4622 scope.go:122] "RemoveContainer" containerID="4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656" Oct 30 17:54:27.669165 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:27.669141 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656\": container with ID starting with 4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656 not found: ID does not exist" containerID="4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656" Oct 30 17:54:27.669246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.669171 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656"} err="failed to get container status \"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656\": rpc error: code = NotFound desc = could not find container \"4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656\": container with ID starting with 4023ec66f2344db8392d226bdfb9cd71ab67664377b0fd0808d06dc29ebc2656 not found: ID does not exist" Oct 30 17:54:27.720189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.720156 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:54:27.730581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.730549 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-5ttkv"] Oct 30 17:54:27.759313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.759284 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:54:27.764659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.764632 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-kz4qj"] Oct 30 17:54:27.799295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.799254 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:54:27.804136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:27.804115 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-043253-openshift-default-58965447bf-pj2jk"] Oct 30 17:54:28.104955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.104923 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l"] Oct 30 17:54:28.105492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105479 4622 cpu_manager.go:414] "containerMap: removing container" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105492 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105515 4622 cpu_manager.go:414] "containerMap: removing container" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105520 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105532 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105538 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105548 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b3db0f0f-479b-4e62-aed1-fca892affc27" containerName="revoke-keys" Oct 30 17:54:28.105552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105553 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3db0f0f-479b-4e62-aed1-fca892affc27" containerName="revoke-keys" Oct 30 17:54:28.105762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105651 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" containerName="istio-proxy" Oct 30 17:54:28.105762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105661 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" containerName="istio-proxy" Oct 30 17:54:28.105762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105668 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" containerName="istio-proxy" Oct 30 17:54:28.105762 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.105675 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3db0f0f-479b-4e62-aed1-fca892affc27" containerName="revoke-keys" Oct 30 17:54:28.110440 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.110420 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.126406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.126210 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:54:28.126406 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.126378 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 17:54:28.129839 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.129675 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l"] Oct 30 17:54:28.173574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.173542 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msvp\" (UniqueName: \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.173744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.173603 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.173744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.173699 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.275143 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.275112 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msvp\" (UniqueName: \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.275649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.275166 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.275649 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.275199 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.275867 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.275843 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.277578 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.277555 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.298023 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.297995 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msvp\" (UniqueName: \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\") pod \"maas-api-revoke-sub-3c0595e6-rk22l\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.375166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.375089 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9392dd6-86d2-498b-8a76-c84e06392bf8" path="/var/lib/kubelet/pods/c9392dd6-86d2-498b-8a76-c84e06392bf8/volumes" Oct 30 17:54:28.375679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.375662 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3" path="/var/lib/kubelet/pods/cc84ae49-4e6e-4360-8c52-eb38ab7f3ab3/volumes" Oct 30 17:54:28.376297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.376280 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e52fa5f6-44d5-4412-b1c5-c98fa850c47b" path="/var/lib/kubelet/pods/e52fa5f6-44d5-4412-b1c5-c98fa850c47b/volumes" Oct 30 17:54:28.437231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.437203 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:28.739750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:28.739723 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l"] Oct 30 17:54:28.739882 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:54:28.739858 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a98425_8801_4055_b8ef_6c49b42b4d7b.slice/crio-c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3 WatchSource:0}: Error finding container c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3: Status 404 returned error can't find the container with id c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3 Oct 30 17:54:29.666804 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:29.666759 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" event={"ID":"17a98425-8801-4055-b8ef-6c49b42b4d7b","Type":"ContainerStarted","Data":"fd3455fa209e69369b870dbaab8670a657bfd55adb8fc059c2fd44e9137bfd58"} Oct 30 17:54:29.667242 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:29.666860 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" event={"ID":"17a98425-8801-4055-b8ef-6c49b42b4d7b","Type":"ContainerStarted","Data":"c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3"} Oct 30 17:54:29.713549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:29.713494 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" podStartSLOduration=1.7134757139999999 podStartE2EDuration="1.713475714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:29.709963853 +0000 UTC m=+2497.487435808" watchObservedRunningTime="2026-10-30 17:54:29.713475714 +0000 UTC m=+2497.490947672" Oct 30 17:54:30.732319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:30.722520 4622 generic.go:356] "Generic (PLEG): container finished" podID="17a98425-8801-4055-b8ef-6c49b42b4d7b" containerID="fd3455fa209e69369b870dbaab8670a657bfd55adb8fc059c2fd44e9137bfd58" exitCode=0 Oct 30 17:54:30.732319 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:30.722783 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" event={"ID":"17a98425-8801-4055-b8ef-6c49b42b4d7b","Type":"ContainerDied","Data":"fd3455fa209e69369b870dbaab8670a657bfd55adb8fc059c2fd44e9137bfd58"} Oct 30 17:54:31.905832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.905807 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:31.936147 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.936121 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\") pod \"17a98425-8801-4055-b8ef-6c49b42b4d7b\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " Oct 30 17:54:31.936322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.936200 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\") pod \"17a98425-8801-4055-b8ef-6c49b42b4d7b\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " Oct 30 17:54:31.936417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.936400 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\" (UniqueName: \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\") pod \"17a98425-8801-4055-b8ef-6c49b42b4d7b\" (UID: \"17a98425-8801-4055-b8ef-6c49b42b4d7b\") " Oct 30 17:54:31.936560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.936540 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca" pod "17a98425-8801-4055-b8ef-6c49b42b4d7b" (UID: "17a98425-8801-4055-b8ef-6c49b42b4d7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:31.936961 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.936944 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:31.938239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.938217 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca" pod "17a98425-8801-4055-b8ef-6c49b42b4d7b" (UID: "17a98425-8801-4055-b8ef-6c49b42b4d7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:54:31.938655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:31.938630 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp" pod "17a98425-8801-4055-b8ef-6c49b42b4d7b" (UID: "17a98425-8801-4055-b8ef-6c49b42b4d7b"). InnerVolumeSpecName "kube-api-access-6msvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:32.038090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:32.038061 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a98425-8801-4055-b8ef-6c49b42b4d7b-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:32.038090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:32.038088 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6msvp\" (UniqueName: \"kubernetes.io/projected/17a98425-8801-4055-b8ef-6c49b42b4d7b-kube-api-access-6msvp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:32.739176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:32.739094 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" Oct 30 17:54:32.739176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:32.739123 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l" event={"ID":"17a98425-8801-4055-b8ef-6c49b42b4d7b","Type":"ContainerDied","Data":"c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3"} Oct 30 17:54:32.739176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:32.739167 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57b587c85beeb9de7558251def03b00306587ae73d2ef3d1ac698cb72fdcab3" Oct 30 17:54:49.837592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.837554 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:54:49.838435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.838411 4622 cpu_manager.go:414] "containerMap: removing container" podUID="17a98425-8801-4055-b8ef-6c49b42b4d7b" containerName="revoke-keys" Oct 30 17:54:49.838435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.838436 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a98425-8801-4055-b8ef-6c49b42b4d7b" containerName="revoke-keys" Oct 30 17:54:49.838674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.838657 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a98425-8801-4055-b8ef-6c49b42b4d7b" containerName="revoke-keys" Oct 30 17:54:49.842557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.842540 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:49.868240 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.868213 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:54:49.875415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.875390 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-4dcdbb\"/\"auth-test-model-4dcdbb-kserve-self-signed-certs\"" Oct 30 17:54:49.875523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.875451 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-4dcdbb\"/\"default-dockercfg-9whhw\"" Oct 30 17:54:49.879104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.879080 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-4dcdbb\"/\"kube-root-ca.crt\"" Oct 30 17:54:49.879217 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:49.879132 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-4dcdbb\"/\"openshift-service-ca.crt\"" Oct 30 17:54:50.006091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006049 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.006401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006103 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.006401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006286 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6m9c\" (UniqueName: \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.006401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006321 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.006539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006443 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.006581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.006560 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109092 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6m9c\" (UniqueName: \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109153 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109290 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109400 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109441 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.109643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.109466 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.113281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.110021 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.113281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.110628 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.113281 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.111631 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.140691 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.140653 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6m9c\" (UniqueName: \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.157284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.142426 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.157284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.143955 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\") pod \"auth-test-model-4dcdbb-kserve-9bff947cc-8db54\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.170349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.160301 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:50.513179 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:54:50.513152 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a04cf3_b810_4abb_bae6_d07abb7bfd4c.slice/crio-7bdbb9eb69b93bfbea40ae4256ba91696d3883d15da13ad9f603a0596753d6e1 WatchSource:0}: Error finding container 7bdbb9eb69b93bfbea40ae4256ba91696d3883d15da13ad9f603a0596753d6e1: Status 404 returned error can't find the container with id 7bdbb9eb69b93bfbea40ae4256ba91696d3883d15da13ad9f603a0596753d6e1 Oct 30 17:54:50.513340 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.513299 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:54:50.877853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.877818 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" event={"ID":"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c","Type":"ContainerStarted","Data":"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb"} Oct 30 17:54:50.878334 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.877873 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" event={"ID":"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c","Type":"ContainerStarted","Data":"7bdbb9eb69b93bfbea40ae4256ba91696d3883d15da13ad9f603a0596753d6e1"} Oct 30 17:54:50.921532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:50.921481 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" podStartSLOduration=1.921468663 podStartE2EDuration="1.921468663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:54:50.91896414 +0000 UTC m=+2518.696436110" watchObservedRunningTime="2026-10-30 17:54:50.921468663 +0000 UTC m=+2518.698940618" Oct 30 17:54:51.169344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:51.163397 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:54:51.685407 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:51.685369 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:54:51.685643 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:51.685617 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" containerID="cri-o://e978c0bee3baa09de390681632c4e8657bdfcbb8b3c1933a989f24ef7a29421e" gracePeriod=30 Oct 30 17:54:54.538209 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:54:54.538174 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:54:55.290371 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:55.290339 4622 patch_prober.go:29] interesting pod/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:54:55.290557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:55.290396 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:54:56.935896 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:56.935855 4622 generic.go:356] "Generic (PLEG): container finished" podID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerID="e978c0bee3baa09de390681632c4e8657bdfcbb8b3c1933a989f24ef7a29421e" exitCode=0 Oct 30 17:54:56.936357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:56.935912 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerDied","Data":"e978c0bee3baa09de390681632c4e8657bdfcbb8b3c1933a989f24ef7a29421e"} Oct 30 17:54:56.936357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:56.935967 4622 scope.go:122] "RemoveContainer" containerID="e727db0e0bea54e53e7b89f0f6ab41a5984812a6a631021e7f0f6a111fd9b43d" Oct 30 17:54:56.990309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:56.990285 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:54:57.012539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012507 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012590 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012642 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012673 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012669 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012714 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012763 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.012833 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012759 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:57.013005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012854 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.013005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012864 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:57.013005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012928 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.013005 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012953 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:57.013223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.012976 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:57.013223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013074 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\") pod \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\" (UID: \"8296a44a-4f3f-4155-84c2-ac7ff45044d8\") " Oct 30 17:54:57.013223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013172 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:54:57.013654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013633 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.013654 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013655 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.013829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013664 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.013829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013674 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.013829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.013683 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.015489 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.015465 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:54:57.015593 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.015508 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "kube-api-access-5qx74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:57.015696 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.015672 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:54:57.015825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.015691 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token" pod "8296a44a-4f3f-4155-84c2-ac7ff45044d8" (UID: "8296a44a-4f3f-4155-84c2-ac7ff45044d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:54:57.114685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.114653 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qx74\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-kube-api-access-5qx74\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.114685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.114679 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.114685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.114688 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.114890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.114697 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8296a44a-4f3f-4155-84c2-ac7ff45044d8-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:54:57.943769 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.943735 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" Oct 30 17:54:57.944225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.943770 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt" event={"ID":"8296a44a-4f3f-4155-84c2-ac7ff45044d8","Type":"ContainerDied","Data":"4a0621833f7a517adf94f3c8a5aafc7b7bd6740f7e95967323d32412be3aafa5"} Oct 30 17:54:57.944225 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:57.943827 4622 scope.go:122] "RemoveContainer" containerID="e978c0bee3baa09de390681632c4e8657bdfcbb8b3c1933a989f24ef7a29421e" Oct 30 17:54:58.013341 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:58.013318 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:54:58.018173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:58.018150 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8d946d-openshift-default-66bf774bf5-2f5zt"] Oct 30 17:54:58.374511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:54:58.374480 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" path="/var/lib/kubelet/pods/8296a44a-4f3f-4155-84c2-ac7ff45044d8/volumes" Oct 30 17:55:00.162165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:00.161920 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:55:00.182331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:00.182301 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:55:00.972449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:00.972417 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc"] Oct 30 17:55:00.978104 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:00.978078 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:55:00.986367 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:00.986335 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bcdac0-kbvkc"] Oct 30 17:55:02.335390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:02.335360 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:55:02.335797 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:02.335573 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" containerID="cri-o://94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19" gracePeriod=30 Oct 30 17:55:02.348388 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:02.348359 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:55:02.348574 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:02.348553 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" containerID="cri-o://bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946" gracePeriod=30 Oct 30 17:55:02.378662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:02.378634 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3db0f0f-479b-4e62-aed1-fca892affc27" path="/var/lib/kubelet/pods/b3db0f0f-479b-4e62-aed1-fca892affc27/volumes" Oct 30 17:55:07.145111 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.145071 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-rzcv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:55:07.145511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.145137 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:55:07.653623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.653597 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:55:07.683786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.683762 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:55:07.791149 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791119 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791185 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.791372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791206 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.791372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791245 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791290 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.791372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791329 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791385 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791414 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791452 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791474 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791477 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791488 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791559 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.791671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791590 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791681 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791727 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791734 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791791 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791833 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791917 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\") pod \"af47d08e-6284-4f32-88d2-9b7bdd32c756\" (UID: \"af47d08e-6284-4f32-88d2-9b7bdd32c756\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.791972 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\") pod \"586b4908-f577-43ab-9d81-5c163ca66412\" (UID: \"586b4908-f577-43ab-9d81-5c163ca66412\") " Oct 30 17:55:07.792126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792104 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:55:07.792576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792467 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.792576 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792472 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.792694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792667 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.792694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792675 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.792694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792685 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.792892 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.792869 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.793131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793104 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793134 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586b4908-f577-43ab-9d81-5c163ca66412-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793149 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793161 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793177 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793191 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793204 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793218 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793231 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.793620 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.793243 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af47d08e-6284-4f32-88d2-9b7bdd32c756-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.794128 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.794107 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.794243 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.794196 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:07.795150 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.795128 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "kube-api-access-tdfg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:07.795513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.795490 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:07.795594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.795548 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:07.797048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.797027 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "kube-api-access-fhsxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:07.797048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.797037 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo" pod "af47d08e-6284-4f32-88d2-9b7bdd32c756" (UID: "af47d08e-6284-4f32-88d2-9b7bdd32c756"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:55:07.797188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.797043 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo" pod "586b4908-f577-43ab-9d81-5c163ca66412" (UID: "586b4908-f577-43ab-9d81-5c163ca66412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:55:07.894986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.894948 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhsxt\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-kube-api-access-fhsxt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.894986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.894979 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586b4908-f577-43ab-9d81-5c163ca66412-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.894986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.894988 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.895203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.894996 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.895203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.895005 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586b4908-f577-43ab-9d81-5c163ca66412-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.895203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.895016 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.895203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.895025 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfg8\" (UniqueName: \"kubernetes.io/projected/586b4908-f577-43ab-9d81-5c163ca66412-kube-api-access-tdfg8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:07.895203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:07.895034 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af47d08e-6284-4f32-88d2-9b7bdd32c756-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:08.025386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.025311 4622 generic.go:356] "Generic (PLEG): container finished" podID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerID="bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946" exitCode=0 Oct 30 17:55:08.025386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.025361 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" Oct 30 17:55:08.025585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.025397 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" event={"ID":"af47d08e-6284-4f32-88d2-9b7bdd32c756","Type":"ContainerDied","Data":"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946"} Oct 30 17:55:08.025585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.025459 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5" event={"ID":"af47d08e-6284-4f32-88d2-9b7bdd32c756","Type":"ContainerDied","Data":"e523e0a61fa36debf4c3cd92e5fa3e6da0b4d262003c26fa6aae469386d1c5ce"} Oct 30 17:55:08.025585 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.025485 4622 scope.go:122] "RemoveContainer" containerID="bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946" Oct 30 17:55:08.027509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.027484 4622 generic.go:356] "Generic (PLEG): container finished" podID="586b4908-f577-43ab-9d81-5c163ca66412" containerID="94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19" exitCode=0 Oct 30 17:55:08.027655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.027536 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" Oct 30 17:55:08.027655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.027567 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerDied","Data":"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19"} Oct 30 17:55:08.027655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.027611 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484" event={"ID":"586b4908-f577-43ab-9d81-5c163ca66412","Type":"ContainerDied","Data":"9c25eb67ba350cb5c8c775808bcaa124c8fd9af023ea3c25c5b5f8a8e7ac0b49"} Oct 30 17:55:08.036459 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.036440 4622 scope.go:122] "RemoveContainer" containerID="bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946" Oct 30 17:55:08.036763 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:08.036741 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946\": container with ID starting with bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946 not found: ID does not exist" containerID="bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946" Oct 30 17:55:08.036861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.036769 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946"} err="failed to get container status \"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946\": rpc error: code = NotFound desc = could not find container \"bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946\": container with ID starting with bb6cce4923e3e29f1609b29341b4f9153b48a7d80232a9871a3986e5c66e7946 not found: ID does not exist" Oct 30 17:55:08.036861 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.036785 4622 scope.go:122] "RemoveContainer" containerID="94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19" Oct 30 17:55:08.046216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.046197 4622 scope.go:122] "RemoveContainer" containerID="4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c" Oct 30 17:55:08.054748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.054733 4622 scope.go:122] "RemoveContainer" containerID="94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19" Oct 30 17:55:08.055035 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:08.055015 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19\": container with ID starting with 94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19 not found: ID does not exist" containerID="94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19" Oct 30 17:55:08.055078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.055046 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19"} err="failed to get container status \"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19\": rpc error: code = NotFound desc = could not find container \"94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19\": container with ID starting with 94b670383260e05c8a6b7894ab5cdcd5bdf3a6b91b59079e55568ad3590d6d19 not found: ID does not exist" Oct 30 17:55:08.055078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.055061 4622 scope.go:122] "RemoveContainer" containerID="4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c" Oct 30 17:55:08.055383 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:08.055363 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c\": container with ID starting with 4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c not found: ID does not exist" containerID="4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c" Oct 30 17:55:08.055446 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.055389 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c"} err="failed to get container status \"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c\": rpc error: code = NotFound desc = could not find container \"4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c\": container with ID starting with 4ae66af3d2a1dea78722113234f2b811855b779827eb056f1960455a8dbe498c not found: ID does not exist" Oct 30 17:55:08.078953 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.078932 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:55:08.085417 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.085397 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-v6484"] Oct 30 17:55:08.122081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.121764 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:55:08.126082 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.126062 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-rzcv5"] Oct 30 17:55:08.375121 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.375092 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586b4908-f577-43ab-9d81-5c163ca66412" path="/var/lib/kubelet/pods/586b4908-f577-43ab-9d81-5c163ca66412/volumes" Oct 30 17:55:08.375689 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:08.375673 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" path="/var/lib/kubelet/pods/af47d08e-6284-4f32-88d2-9b7bdd32c756/volumes" Oct 30 17:55:11.365436 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365395 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 17:55:11.365938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365925 4622 cpu_manager.go:414] "containerMap: removing container" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365940 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365951 4622 cpu_manager.go:414] "containerMap: removing container" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365957 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365983 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365988 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.365997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.365995 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.366178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.366000 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.366178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.366090 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:11.366178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.366098 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.366178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.366105 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8296a44a-4f3f-4155-84c2-ac7ff45044d8" containerName="istio-proxy" Oct 30 17:55:11.366178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.366116 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="af47d08e-6284-4f32-88d2-9b7bdd32c756" containerName="istio-proxy" Oct 30 17:55:11.371368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.371343 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.399561 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.399534 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-d207f8-serving-cert\"" Oct 30 17:55:11.404057 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.404026 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 17:55:11.446320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.446284 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.446476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.446443 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.446528 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.446509 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8885\" (UniqueName: \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.446655 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.446636 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.547944 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.547909 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.548135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.547963 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8885\" (UniqueName: \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.548135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.548018 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.548135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.548097 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.548625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.548591 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.550515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.550493 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.550610 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.550552 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.571206 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.571185 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8885\" (UniqueName: \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\") pod \"maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.685128 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.685040 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:11.996519 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:55:11.996487 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10630542_7a98_4003_a343_2498ff3cdd99.slice/crio-33bce92a5065b0b6778dc00a67f765a40c57d5e52b736910ab166b1b3b17d90a WatchSource:0}: Error finding container 33bce92a5065b0b6778dc00a67f765a40c57d5e52b736910ab166b1b3b17d90a: Status 404 returned error can't find the container with id 33bce92a5065b0b6778dc00a67f765a40c57d5e52b736910ab166b1b3b17d90a Oct 30 17:55:11.996897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.996876 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 17:55:11.997884 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:11.997867 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:55:12.056152 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:12.056114 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" event={"ID":"10630542-7a98-4003-a343-2498ff3cdd99","Type":"ContainerStarted","Data":"33bce92a5065b0b6778dc00a67f765a40c57d5e52b736910ab166b1b3b17d90a"} Oct 30 17:55:13.064375 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:13.064342 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" event={"ID":"10630542-7a98-4003-a343-2498ff3cdd99","Type":"ContainerStarted","Data":"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f"} Oct 30 17:55:13.064872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:13.064454 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:13.109730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:13.109686 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" podStartSLOduration=1.629803147 podStartE2EDuration="2.109671094s" totalImagesPullingTime="479.867947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:13.107785466 +0000 UTC m=+2540.885257424" watchObservedRunningTime="2026-10-30 17:55:13.109671094 +0000 UTC m=+2540.887143072" Oct 30 17:55:19.077860 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:19.077830 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 17:55:23.675011 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:23.674923 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:55:24.537337 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:24.537297 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:55:27.703846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.703810 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:55:27.704764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.704739 4622 cpu_manager.go:414] "containerMap: removing container" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:27.704764 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.704765 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:27.704919 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.704906 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="586b4908-f577-43ab-9d81-5c163ca66412" containerName="istio-proxy" Oct 30 17:55:27.709357 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.709334 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.736102 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.736072 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:55:27.742990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.742953 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a17354\"/\"kube-root-ca.crt\"" Oct 30 17:55:27.742990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.742962 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-a17354\"/\"openshift-service-ca.crt\"" Oct 30 17:55:27.743186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.743008 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a17354\"/\"auth-test-model-a17354-kserve-self-signed-certs\"" Oct 30 17:55:27.743186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.743083 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-a17354\"/\"default-dockercfg-b8rzl\"" Oct 30 17:55:27.861868 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.861830 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.862034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.861889 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.862034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.862016 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.862110 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.862046 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwjr\" (UniqueName: \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.862201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.862183 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.862368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.862352 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.964302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.964197 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.964565 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.964542 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.964809 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.964791 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.966753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.966726 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwjr\" (UniqueName: \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.967392 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.967370 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.968000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.967638 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.968727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.965882 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.968872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.965641 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.969050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.969029 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.982277 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.971762 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:27.994843 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:27.994803 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:28.017317 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:28.001889 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwjr\" (UniqueName: \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\") pod \"auth-test-model-a17354-kserve-7f7995f469-4hwrd\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:28.024199 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:28.023848 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:28.335494 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:28.335465 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:55:28.335912 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:55:28.335884 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f36a23d_9689_4b26_aeab_f22591389a76.slice/crio-e720737e9555d4b7d747d1b4d89ae1c5f22f6d12297bc406cae46e0f317a2eeb WatchSource:0}: Error finding container e720737e9555d4b7d747d1b4d89ae1c5f22f6d12297bc406cae46e0f317a2eeb: Status 404 returned error can't find the container with id e720737e9555d4b7d747d1b4d89ae1c5f22f6d12297bc406cae46e0f317a2eeb Oct 30 17:55:29.186508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:29.186474 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" event={"ID":"9f36a23d-9689-4b26-aeab-f22591389a76","Type":"ContainerStarted","Data":"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728"} Oct 30 17:55:29.186894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:29.186523 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" event={"ID":"9f36a23d-9689-4b26-aeab-f22591389a76","Type":"ContainerStarted","Data":"e720737e9555d4b7d747d1b4d89ae1c5f22f6d12297bc406cae46e0f317a2eeb"} Oct 30 17:55:29.238058 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:29.238005 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" podStartSLOduration=2.237987615 podStartE2EDuration="2.237987615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:55:29.23459969 +0000 UTC m=+2557.012071646" watchObservedRunningTime="2026-10-30 17:55:29.237987615 +0000 UTC m=+2557.015459574" Oct 30 17:55:30.024928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:30.024900 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:38.025310 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:38.025257 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:38.037563 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:38.037539 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:38.264029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:38.264001 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:55:38.701584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:38.701534 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" podUID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" containerName="main" containerID="cri-o://960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a" gracePeriod=45 Oct 30 17:55:38.981992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:38.981968 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:55:39.020285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020240 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020409 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020338 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020435 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\" (UniqueName: \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020532 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020519 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020541 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020435 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:39.020674 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020591 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\") pod \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\" (UID: \"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7\") " Oct 30 17:55:39.020864 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020796 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:39.020950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.020908 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:39.021076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.021057 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.021132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.021085 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.021132 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.021098 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.023885 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.023844 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:55:39.024338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.024314 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "kube-api-access-5zj2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:55:39.024439 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.024389 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm" pod "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" (UID: "51418bab-0a0b-4bc7-a08f-1ac09ee8baf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:55:39.122496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.122463 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.122496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.122489 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zj2c\" (UniqueName: \"kubernetes.io/projected/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-kube-api-access-5zj2c\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.122496 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.122499 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:55:39.261645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.261564 4622 generic.go:356] "Generic (PLEG): container finished" podID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" containerID="960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a" exitCode=0 Oct 30 17:55:39.261645 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.261623 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" Oct 30 17:55:39.261827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.261651 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" event={"ID":"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7","Type":"ContainerDied","Data":"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a"} Oct 30 17:55:39.261827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.261700 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx" event={"ID":"51418bab-0a0b-4bc7-a08f-1ac09ee8baf7","Type":"ContainerDied","Data":"2c5efd929b0f31569ad39d51aa468819947789924e36fcee3a8f88725c5a526b"} Oct 30 17:55:39.261827 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.261723 4622 scope.go:122] "RemoveContainer" containerID="960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a" Oct 30 17:55:39.273378 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.273358 4622 scope.go:122] "RemoveContainer" containerID="960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a" Oct 30 17:55:39.275732 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:39.275703 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a\": container with ID starting with 960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a not found: ID does not exist" containerID="960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a" Oct 30 17:55:39.275834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.275739 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a"} err="failed to get container status \"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a\": rpc error: code = NotFound desc = could not find container \"960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a\": container with ID starting with 960d16d70cf8f7109e18270972a06cbc0a0ce97bf1c5851ee106d9905d7a554a not found: ID does not exist" Oct 30 17:55:39.313641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.313573 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:55:39.325679 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:39.325656 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/auth-test-model-d53ec1-kserve-7b6c97d4b9-xz6cx"] Oct 30 17:55:40.374831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:40.374798 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" path="/var/lib/kubelet/pods/51418bab-0a0b-4bc7-a08f-1ac09ee8baf7/volumes" Oct 30 17:55:54.622179 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:55:54.622140 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:55:59.141830 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.141784 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:55:59.142723 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.142698 4622 cpu_manager.go:414] "containerMap: removing container" podUID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" containerName="main" Oct 30 17:55:59.142835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.142724 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" containerName="main" Oct 30 17:55:59.142920 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.142907 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="51418bab-0a0b-4bc7-a08f-1ac09ee8baf7" containerName="main" Oct 30 17:55:59.148487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.148469 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.166197 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.166172 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:55:59.174460 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.174432 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"openshift-service-ca.crt\"" Oct 30 17:55:59.174575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.174518 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"default-dockercfg-kjgn4\"" Oct 30 17:55:59.175302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.175281 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"auth-test-model-a23c98-kserve-self-signed-certs\"" Oct 30 17:55:59.263449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.263416 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"kube-root-ca.crt\"" Oct 30 17:55:59.308433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308401 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.308433 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308432 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.308659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308469 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.308731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308706 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.308780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308756 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fsd\" (UniqueName: \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.308975 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.308934 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.409994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.409901 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410021 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410050 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410388 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410410 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410402 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410542 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410496 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410590 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.410662 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.410624 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fsd\" (UniqueName: \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.411034 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.411006 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.412664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.412632 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.414051 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.414020 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.439781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.439714 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fsd\" (UniqueName: \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\") pod \"auth-test-model-a23c98-kserve-96cd58b97-l79fp\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.463588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.463546 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:55:59.823644 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:55:59.823615 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:55:59.824715 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:55:59.824684 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dbf3931_fd28_4a21_9f7c_f75687666150.slice/crio-00dde96144453680be5773e62844e67c98bebb8b4ad6ce9cc693e19f060c15c8 WatchSource:0}: Error finding container 00dde96144453680be5773e62844e67c98bebb8b4ad6ce9cc693e19f060c15c8: Status 404 returned error can't find the container with id 00dde96144453680be5773e62844e67c98bebb8b4ad6ce9cc693e19f060c15c8 Oct 30 17:56:00.417338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:00.417296 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" event={"ID":"0dbf3931-fd28-4a21-9f7c-f75687666150","Type":"ContainerStarted","Data":"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04"} Oct 30 17:56:00.417802 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:00.417352 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" event={"ID":"0dbf3931-fd28-4a21-9f7c-f75687666150","Type":"ContainerStarted","Data":"00dde96144453680be5773e62844e67c98bebb8b4ad6ce9cc693e19f060c15c8"} Oct 30 17:56:00.464195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:00.464119 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" podStartSLOduration=1.4641011800000001 podStartE2EDuration="1.46410118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:00.462045819 +0000 UTC m=+2588.239517776" watchObservedRunningTime="2026-10-30 17:56:00.46410118 +0000 UTC m=+2588.241573135" Oct 30 17:56:00.464394 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:00.464233 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:56:09.464770 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:09.464738 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:56:09.477091 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:09.477062 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:56:09.497059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:09.497028 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:56:19.354193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:19.354156 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:56:24.534210 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:56:24.534175 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:56:34.375865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.375804 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" podUID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" containerName="main" containerID="cri-o://24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e" gracePeriod=45 Oct 30 17:56:34.637290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.637214 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:56:34.655019 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.654994 4622 generic.go:356] "Generic (PLEG): container finished" podID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" containerID="24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e" exitCode=0 Oct 30 17:56:34.655165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.655035 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" Oct 30 17:56:34.655165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.655075 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" event={"ID":"2d967223-c9d6-46c5-ab6d-5b72fbd0a778","Type":"ContainerDied","Data":"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e"} Oct 30 17:56:34.655165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.655114 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8" event={"ID":"2d967223-c9d6-46c5-ab6d-5b72fbd0a778","Type":"ContainerDied","Data":"56d1d2a129487ebaf50cdce4f5400c08705d86086f0356544bac6931d1e30649"} Oct 30 17:56:34.655165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.655134 4622 scope.go:122] "RemoveContainer" containerID="24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e" Oct 30 17:56:34.668159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.668146 4622 scope.go:122] "RemoveContainer" containerID="24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e" Oct 30 17:56:34.668777 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:56:34.668747 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e\": container with ID starting with 24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e not found: ID does not exist" containerID="24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e" Oct 30 17:56:34.668891 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.668786 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e"} err="failed to get container status \"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e\": rpc error: code = NotFound desc = could not find container \"24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e\": container with ID starting with 24c25f58a35247806319c0a932f7e5ac4206d48719dbfaa8ccd5897cd591899e not found: ID does not exist" Oct 30 17:56:34.786342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786300 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786348 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786377 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786399 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786636 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786649 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:34.786707 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786694 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\") pod \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\" (UID: \"2d967223-c9d6-46c5-ab6d-5b72fbd0a778\") " Oct 30 17:56:34.786862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786707 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:34.786973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.786959 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:34.787216 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.787200 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:34.787295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.787219 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:34.787295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.787227 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:34.788751 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.788728 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "kube-api-access-bgldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:34.788966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.788943 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:56:34.789070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.788959 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm" pod "2d967223-c9d6-46c5-ab6d-5b72fbd0a778" (UID: "2d967223-c9d6-46c5-ab6d-5b72fbd0a778"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:34.887940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.887874 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgldj\" (UniqueName: \"kubernetes.io/projected/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-kube-api-access-bgldj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:34.887940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.887899 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:34.887940 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:34.887910 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d967223-c9d6-46c5-ab6d-5b72fbd0a778-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:35.004595 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.004565 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:56:35.017171 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.017141 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-ce0480/auth-test-model-ce0480-kserve-588456cd57-f4nq8"] Oct 30 17:56:35.020680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.020650 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:56:35.021231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.021213 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" containerName="main" Oct 30 17:56:35.021344 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.021232 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" containerName="main" Oct 30 17:56:35.021449 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.021435 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" containerName="main" Oct 30 17:56:35.025195 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.025171 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.045331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.040573 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-016f1d\"/\"openshift-service-ca.crt\"" Oct 30 17:56:35.045331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.040709 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-016f1d\"/\"kube-root-ca.crt\"" Oct 30 17:56:35.045331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.040576 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-016f1d\"/\"default-dockercfg-vf56l\"" Oct 30 17:56:35.045331 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.041235 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-016f1d\"/\"auth-test-model-016f1d-kserve-self-signed-certs\"" Oct 30 17:56:35.051350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.051322 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:56:35.089895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.089864 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.090007 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.089917 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmt8v\" (UniqueName: \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.090059 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.090039 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.090120 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.090105 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.090166 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.090155 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.090339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.090312 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192092 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.191994 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192096 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192151 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192194 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmt8v\" (UniqueName: \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192298 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192238 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192329 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192448 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192534 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192504 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.192705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.192660 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.194697 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.194666 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.194819 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.194773 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.215329 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.215306 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmt8v\" (UniqueName: \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\") pod \"auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.348829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.348489 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:35.666605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:35.666579 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:56:35.667396 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:56:35.667366 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod575ef9bf_bb2a_41c8_90ea_483224bd5ef8.slice/crio-2dd475a8e0ba38f7d429b1c6398ff942926e626bcd61c32b4081af8f65128559 WatchSource:0}: Error finding container 2dd475a8e0ba38f7d429b1c6398ff942926e626bcd61c32b4081af8f65128559: Status 404 returned error can't find the container with id 2dd475a8e0ba38f7d429b1c6398ff942926e626bcd61c32b4081af8f65128559 Oct 30 17:56:36.374736 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:36.374702 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d967223-c9d6-46c5-ab6d-5b72fbd0a778" path="/var/lib/kubelet/pods/2d967223-c9d6-46c5-ab6d-5b72fbd0a778/volumes" Oct 30 17:56:36.676992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:36.676912 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" event={"ID":"575ef9bf-bb2a-41c8-90ea-483224bd5ef8","Type":"ContainerStarted","Data":"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea"} Oct 30 17:56:36.676992 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:36.676959 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" event={"ID":"575ef9bf-bb2a-41c8-90ea-483224bd5ef8","Type":"ContainerStarted","Data":"2dd475a8e0ba38f7d429b1c6398ff942926e626bcd61c32b4081af8f65128559"} Oct 30 17:56:36.718191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:36.718138 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" podStartSLOduration=2.718120022 podStartE2EDuration="2.718120022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:36.717220533 +0000 UTC m=+2624.494692511" watchObservedRunningTime="2026-10-30 17:56:36.718120022 +0000 UTC m=+2624.495591979" Oct 30 17:56:37.349677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:37.349648 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:43.537955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:43.537916 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:56:45.350077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:45.350044 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:45.380857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:45.380823 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:45.771248 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:45.771164 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:56:54.534644 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:56:54.534567 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:56:55.429903 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.429869 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 17:56:55.436237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.436214 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.463415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.463390 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 17:56:55.471556 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.471529 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"kube-root-ca.crt\"" Oct 30 17:56:55.471671 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.471569 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"sub-test-model-d53ec1-kserve-self-signed-certs\"" Oct 30 17:56:55.471741 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.471718 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"openshift-service-ca.crt\"" Oct 30 17:56:55.471803 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.471791 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d53ec1\"/\"default-dockercfg-69tbc\"" Oct 30 17:56:55.567078 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567040 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.567599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567254 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.567599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567423 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.567599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567473 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4tq\" (UniqueName: \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.567599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567495 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.567895 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.567611 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.670239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.670199 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.671228 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.671201 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.671957 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.671924 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4tq\" (UniqueName: \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.672530 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.672499 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.673126 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.673099 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.673223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.673148 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.678309 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.670988 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.678772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.678713 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.679241 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.679223 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.695622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.695559 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.703135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.702540 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.703923 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.703899 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4tq\" (UniqueName: \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\") pod \"sub-test-model-d53ec1-kserve-75c9f46f79-r57cd\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:55.752306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:55.751201 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:56.074752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.074721 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 17:56:56.075466 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:56:56.075437 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc23ea3f_5df7_4f79_879c_3b5b7d8d413e.slice/crio-5662d273addc87267e2f60d8afc01c9454efd1485b1834f16359bccefa337ecf WatchSource:0}: Error finding container 5662d273addc87267e2f60d8afc01c9454efd1485b1834f16359bccefa337ecf: Status 404 returned error can't find the container with id 5662d273addc87267e2f60d8afc01c9454efd1485b1834f16359bccefa337ecf Oct 30 17:56:56.681889 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.681858 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 17:56:56.685971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.685952 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.700682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.700663 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a23c98-openshift-default-dockercfg-g2m5h\"" Oct 30 17:56:56.718676 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.718652 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 17:56:56.793555 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793525 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.793757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793560 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.793757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793596 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.793757 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793665 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgzv\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.793937 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793832 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.793987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.793944 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.794037 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.794014 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.794093 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.794071 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.794155 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.794129 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.842219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.842183 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" event={"ID":"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e","Type":"ContainerStarted","Data":"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d"} Oct 30 17:56:56.842414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.842226 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" event={"ID":"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e","Type":"ContainerStarted","Data":"5662d273addc87267e2f60d8afc01c9454efd1485b1834f16359bccefa337ecf"} Oct 30 17:56:56.887476 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.887425 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" podStartSLOduration=1.8874133309999999 podStartE2EDuration="1.887413331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:56.885221062 +0000 UTC m=+2644.662693017" watchObservedRunningTime="2026-10-30 17:56:56.887413331 +0000 UTC m=+2644.664885286" Oct 30 17:56:56.895606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895575 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895685 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895737 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895718 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895769 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895820 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895845 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895876 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.895914 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.895900 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgzv\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.896190 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.896147 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.896453 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.896423 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.896567 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.896478 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.896622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.896586 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.896622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.896604 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.897991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.897968 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.898365 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.898346 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.918848 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.918827 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:56.919042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:56.919025 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgzv\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\") pod \"e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:57.005336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.004344 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:57.326216 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:56:57.326185 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31ffeb5_1de8_4da9_9140_5ed5cd104eb8.slice/crio-3eb16950b3a26fb536c3996b1159a14205be1eaf24a906bca7ce7fb1148d0941 WatchSource:0}: Error finding container 3eb16950b3a26fb536c3996b1159a14205be1eaf24a906bca7ce7fb1148d0941: Status 404 returned error can't find the container with id 3eb16950b3a26fb536c3996b1159a14205be1eaf24a906bca7ce7fb1148d0941 Oct 30 17:56:57.326339 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.326306 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 17:56:57.328750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.328713 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:56:57.328852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.328782 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:56:57.328852 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.328835 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:56:57.752192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.752121 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:56:57.850462 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.850424 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" event={"ID":"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8","Type":"ContainerStarted","Data":"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3"} Oct 30 17:56:57.850633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.850481 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" event={"ID":"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8","Type":"ContainerStarted","Data":"3eb16950b3a26fb536c3996b1159a14205be1eaf24a906bca7ce7fb1148d0941"} Oct 30 17:56:57.908527 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:57.908477 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podStartSLOduration=1.9084636160000001 podStartE2EDuration="1.908463616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:57.906503641 +0000 UTC m=+2645.683975598" watchObservedRunningTime="2026-10-30 17:56:57.908463616 +0000 UTC m=+2645.685935571" Oct 30 17:56:58.005774 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.005709 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:56:58.007338 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.007318 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:56:58.007397 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.007359 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:56:58.560903 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.560852 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" podUID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" containerName="main" containerID="cri-o://0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1" gracePeriod=45 Oct 30 17:56:58.847589 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.847567 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:56:58.868233 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.868203 4622 generic.go:356] "Generic (PLEG): container finished" podID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" containerID="0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1" exitCode=0 Oct 30 17:56:58.868401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.868296 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" Oct 30 17:56:58.868401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.868314 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" event={"ID":"bd8f5a05-f021-4fcb-92b0-55d66e41334b","Type":"ContainerDied","Data":"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1"} Oct 30 17:56:58.868401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.868359 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l" event={"ID":"bd8f5a05-f021-4fcb-92b0-55d66e41334b","Type":"ContainerDied","Data":"3917ac0883f4e7d591148e6e4e634e46ba219487d49e20b5c1b0ccbe673c807b"} Oct 30 17:56:58.868401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.868380 4622 scope.go:122] "RemoveContainer" containerID="0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1" Oct 30 17:56:58.878169 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.878150 4622 scope.go:122] "RemoveContainer" containerID="0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1" Oct 30 17:56:58.878445 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:56:58.878423 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1\": container with ID starting with 0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1 not found: ID does not exist" containerID="0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1" Oct 30 17:56:58.878531 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.878452 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1"} err="failed to get container status \"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1\": rpc error: code = NotFound desc = could not find container \"0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1\": container with ID starting with 0c8f16859f13ed75040fb800f41203e4214210b72e7de63ca826be499694cee1 not found: ID does not exist" Oct 30 17:56:58.946219 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946192 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946320 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\" (UniqueName: \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946355 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946378 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946453 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:58.946582 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946530 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946644 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\") pod \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\" (UID: \"bd8f5a05-f021-4fcb-92b0-55d66e41334b\") " Oct 30 17:56:58.946735 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946643 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:58.946825 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.946805 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:58.947181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.947161 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:58.947308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.947209 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:58.947308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.947222 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:58.948683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.948661 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:56:58.948786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.948665 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:56:58.948786 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:58.948717 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq" pod "bd8f5a05-f021-4fcb-92b0-55d66e41334b" (UID: "bd8f5a05-f021-4fcb-92b0-55d66e41334b"). InnerVolumeSpecName "kube-api-access-ldqjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:56:59.005098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.005069 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:56:59.005227 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.005129 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:56:59.048413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.048379 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqjq\" (UniqueName: \"kubernetes.io/projected/bd8f5a05-f021-4fcb-92b0-55d66e41334b-kube-api-access-ldqjq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:59.048413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.048407 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd8f5a05-f021-4fcb-92b0-55d66e41334b-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:59.048413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.048419 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd8f5a05-f021-4fcb-92b0-55d66e41334b-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:56:59.214599 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.214570 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:56:59.218029 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:56:59.218006 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-01b925/auth-test-model-01b925-kserve-6b5bcb9bd5-q4g9l"] Oct 30 17:57:00.005435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:00.005399 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:57:00.005835 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:00.005447 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:57:00.375177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:00.375134 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" path="/var/lib/kubelet/pods/bd8f5a05-f021-4fcb-92b0-55d66e41334b/volumes" Oct 30 17:57:01.005454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:01.005423 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:57:01.005894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:01.005476 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:57:02.005192 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:02.005154 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:57:02.005391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:02.005210 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:57:03.005829 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:03.005794 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:57:03.006222 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:03.005847 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:57:04.004897 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:04.004861 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:57:04.005083 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:04.004917 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 17:57:05.111550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.111518 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:57:05.112515 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.112482 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:57:05.113295 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.113245 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 17:57:05.751872 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.751839 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:57:05.764437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.764414 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:57:05.929973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:05.929944 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 17:57:12.865743 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.865709 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 17:57:12.866509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.866488 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" containerName="main" Oct 30 17:57:12.866509 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.866509 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" containerName="main" Oct 30 17:57:12.866683 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.866651 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8f5a05-f021-4fcb-92b0-55d66e41334b" containerName="main" Oct 30 17:57:12.870156 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.870136 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.900098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.900054 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"openshift-service-ca.crt\"" Oct 30 17:57:12.900098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.900054 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"default-dockercfg-lskph\"" Oct 30 17:57:12.900098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.900055 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"sub-test-model-d207f8-kserve-self-signed-certs\"" Oct 30 17:57:12.900435 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.900130 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"kube-root-ca.crt\"" Oct 30 17:57:12.903870 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.903844 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 17:57:12.950560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950530 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.950744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950566 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9z4\" (UniqueName: \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.950744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950604 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.950744 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950673 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.950931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950799 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:12.950931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:12.950829 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052310 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052351 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9z4\" (UniqueName: \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052391 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052410 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052433 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052622 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052463 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052862 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052799 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.052931 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.052871 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.053065 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.053040 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.054847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.054819 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.055560 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.055532 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.076403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.076369 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9z4\" (UniqueName: \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\") pod \"sub-test-model-d207f8-kserve-54ffcb9559-4rdf4\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.183685 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.183592 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:13.523109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.523070 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 17:57:13.523281 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:57:13.523073 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41b331a9_ceb3_4e1d_9748_0ee1fcacb6fe.slice/crio-0d0fdf02f41f83333df862cd8f425aae5bbbba6d85da193b915ddb68e65a0301 WatchSource:0}: Error finding container 0d0fdf02f41f83333df862cd8f425aae5bbbba6d85da193b915ddb68e65a0301: Status 404 returned error can't find the container with id 0d0fdf02f41f83333df862cd8f425aae5bbbba6d85da193b915ddb68e65a0301 Oct 30 17:57:13.993592 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.993559 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" event={"ID":"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe","Type":"ContainerStarted","Data":"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0"} Oct 30 17:57:13.994123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:13.993604 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" event={"ID":"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe","Type":"ContainerStarted","Data":"0d0fdf02f41f83333df862cd8f425aae5bbbba6d85da193b915ddb68e65a0301"} Oct 30 17:57:14.035990 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:14.035921 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" podStartSLOduration=2.035903397 podStartE2EDuration="2.035903397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:14.034569336 +0000 UTC m=+2661.812041292" watchObservedRunningTime="2026-10-30 17:57:14.035903397 +0000 UTC m=+2661.813375355" Oct 30 17:57:14.183791 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:14.183765 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:22.739618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:22.739586 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:57:23.184036 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:23.183993 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:23.197137 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:23.197109 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:24.087109 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:24.087078 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 17:57:24.533989 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:57:24.533955 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:57:36.641285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.637773 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 17:57:36.646987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.646717 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.673328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.673299 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 17:57:36.738379 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.738353 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-738845-openshift-default-dockercfg-5lqwd\"" Oct 30 17:57:36.774513 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774485 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774520 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774545 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774731 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774779 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774865 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8crl\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774889 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.774948 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.774909 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.775116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.775012 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.876237 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876194 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.876478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876406 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8crl\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.876478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876448 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.876648 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876479 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877017 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876988 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877297 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877243 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.876556 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877377 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877415 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877523 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877449 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877635 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877608 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877784 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877756 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.877879 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.877849 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.878425 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.878401 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.879245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.879219 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.879750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.879728 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.899865 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.899811 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8crl\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.899985 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.899891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\") pod \"e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:36.973857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:36.973829 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:37.307847 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:57:37.307807 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefde37f6_504c_4dd5_9764_65910dc63ca2.slice/crio-d00a2e804369d9a2d8e38cd9dcb3a7627a15c72828a3bb363352b6092cd63735 WatchSource:0}: Error finding container d00a2e804369d9a2d8e38cd9dcb3a7627a15c72828a3bb363352b6092cd63735: Status 404 returned error can't find the container with id d00a2e804369d9a2d8e38cd9dcb3a7627a15c72828a3bb363352b6092cd63735 Oct 30 17:57:37.308478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:37.308450 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 17:57:37.311154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:37.311116 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:57:37.311253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:37.311199 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:57:37.311253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:37.311242 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:57:37.765328 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:37.765066 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" podUID="9f36a23d-9689-4b26-aeab-f22591389a76" containerName="main" containerID="cri-o://5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728" gracePeriod=45 Oct 30 17:57:38.051928 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.051901 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:57:38.174979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.174947 4622 generic.go:356] "Generic (PLEG): container finished" podID="9f36a23d-9689-4b26-aeab-f22591389a76" containerID="5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728" exitCode=0 Oct 30 17:57:38.175160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.174999 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" Oct 30 17:57:38.175160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.175041 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" event={"ID":"9f36a23d-9689-4b26-aeab-f22591389a76","Type":"ContainerDied","Data":"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728"} Oct 30 17:57:38.175160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.175094 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd" event={"ID":"9f36a23d-9689-4b26-aeab-f22591389a76","Type":"ContainerDied","Data":"e720737e9555d4b7d747d1b4d89ae1c5f22f6d12297bc406cae46e0f317a2eeb"} Oct 30 17:57:38.175160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.175119 4622 scope.go:122] "RemoveContainer" containerID="5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728" Oct 30 17:57:38.177006 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.176971 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerStarted","Data":"b9f5a9a05d9d4987ea8c2e4488ec968c14afa0c27eebe4b3db9cc6568e44ed88"} Oct 30 17:57:38.177138 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.177026 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerStarted","Data":"d00a2e804369d9a2d8e38cd9dcb3a7627a15c72828a3bb363352b6092cd63735"} Oct 30 17:57:38.185579 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.185553 4622 scope.go:122] "RemoveContainer" containerID="5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728" Oct 30 17:57:38.185857 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:57:38.185840 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728\": container with ID starting with 5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728 not found: ID does not exist" containerID="5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728" Oct 30 17:57:38.185907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.185867 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728"} err="failed to get container status \"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728\": rpc error: code = NotFound desc = could not find container \"5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728\": container with ID starting with 5be10d64eca798e27cf5b7f36076cc040c36fbe2350b611c53f797e4f9c5b728 not found: ID does not exist" Oct 30 17:57:38.201084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201054 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\" (UniqueName: \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201097 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201193 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201130 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201343 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201325 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201451 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201426 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:38.201535 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201437 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201574 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:38.201608 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201588 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\") pod \"9f36a23d-9689-4b26-aeab-f22591389a76\" (UID: \"9f36a23d-9689-4b26-aeab-f22591389a76\") " Oct 30 17:57:38.201826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.201808 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:38.202246 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.202224 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.202355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.202252 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.202355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.202301 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.203537 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.203508 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:38.203634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.203558 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "kube-api-access-qmwjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:38.203634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.203628 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs" pod "9f36a23d-9689-4b26-aeab-f22591389a76" (UID: "9f36a23d-9689-4b26-aeab-f22591389a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:38.238882 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.238834 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podStartSLOduration=2.238821756 podStartE2EDuration="2.238821756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:38.236147343 +0000 UTC m=+2686.013619299" watchObservedRunningTime="2026-10-30 17:57:38.238821756 +0000 UTC m=+2686.016293710" Oct 30 17:57:38.304090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.304064 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwjr\" (UniqueName: \"kubernetes.io/projected/9f36a23d-9689-4b26-aeab-f22591389a76-kube-api-access-qmwjr\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.304090 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.304093 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f36a23d-9689-4b26-aeab-f22591389a76-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.304360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.304106 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f36a23d-9689-4b26-aeab-f22591389a76-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:38.329414 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.327091 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:57:38.547362 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.547327 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:57:38.550963 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.550939 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-a17354/auth-test-model-a17354-kserve-7f7995f469-4hwrd"] Oct 30 17:57:38.974665 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.974622 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:38.982603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:38.982567 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:39.187713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:39.187683 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:39.188918 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:39.188895 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:57:40.376785 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:40.376751 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f36a23d-9689-4b26-aeab-f22591389a76" path="/var/lib/kubelet/pods/9f36a23d-9689-4b26-aeab-f22591389a76/volumes" Oct 30 17:57:47.567841 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:47.567800 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:57:53.355477 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.355424 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" podUID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" containerName="main" containerID="cri-o://4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea" gracePeriod=45 Oct 30 17:57:53.626173 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.626147 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:57:53.718929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.718902 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.718953 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719006 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\" (UniqueName: \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719075 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719203 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719256 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:53.719457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719319 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\" (UniqueName: \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\") pod \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\" (UID: \"575ef9bf-bb2a-41c8-90ea-483224bd5ef8\") " Oct 30 17:57:53.719457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719375 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:53.719457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719430 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:53.719858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719828 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:53.719858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719847 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:53.719858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.719855 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:53.721301 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.721234 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:53.721605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.721583 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:53.721605 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.721593 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v" pod "575ef9bf-bb2a-41c8-90ea-483224bd5ef8" (UID: "575ef9bf-bb2a-41c8-90ea-483224bd5ef8"). InnerVolumeSpecName "kube-api-access-lmt8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:53.821487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.821454 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:53.821487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.821481 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmt8v\" (UniqueName: \"kubernetes.io/projected/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-kube-api-access-lmt8v\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:53.821487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:53.821492 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575ef9bf-bb2a-41c8-90ea-483224bd5ef8-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:57:54.321323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.321249 4622 generic.go:356] "Generic (PLEG): container finished" podID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" containerID="4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea" exitCode=0 Oct 30 17:57:54.321323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.321302 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" event={"ID":"575ef9bf-bb2a-41c8-90ea-483224bd5ef8","Type":"ContainerDied","Data":"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea"} Oct 30 17:57:54.321549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.321347 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" event={"ID":"575ef9bf-bb2a-41c8-90ea-483224bd5ef8","Type":"ContainerDied","Data":"2dd475a8e0ba38f7d429b1c6398ff942926e626bcd61c32b4081af8f65128559"} Oct 30 17:57:54.321549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.321312 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf" Oct 30 17:57:54.321549 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.321364 4622 scope.go:122] "RemoveContainer" containerID="4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea" Oct 30 17:57:54.332443 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.332425 4622 scope.go:122] "RemoveContainer" containerID="4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea" Oct 30 17:57:54.332690 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:57:54.332670 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea\": container with ID starting with 4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea not found: ID does not exist" containerID="4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea" Oct 30 17:57:54.332767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.332700 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea"} err="failed to get container status \"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea\": rpc error: code = NotFound desc = could not find container \"4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea\": container with ID starting with 4dccea3e5ba02348a34df3712bfa8ec6c5d13b8c5ae272d1bc3af42f43854bea not found: ID does not exist" Oct 30 17:57:54.375859 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.375833 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:57:54.385850 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:54.385829 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-016f1d/auth-test-model-016f1d-kserve-5cc96c9bdf-ppxcf"] Oct 30 17:57:54.534032 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:57:54.533995 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:57:56.375564 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:57:56.375529 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" path="/var/lib/kubelet/pods/575ef9bf-bb2a-41c8-90ea-483224bd5ef8/volumes" Oct 30 17:58:02.591584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.591525 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" podUID="e6338952-93f9-4634-b015-3ee549712d4d" containerName="main" containerID="cri-o://064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad" gracePeriod=45 Oct 30 17:58:02.883706 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.883671 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:58:02.935202 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935172 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935297 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935342 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935395 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935390 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935460 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:02.935587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935557 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935590 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\" (UniqueName: \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\") pod \"e6338952-93f9-4634-b015-3ee549712d4d\" (UID: \"e6338952-93f9-4634-b015-3ee549712d4d\") " Oct 30 17:58:02.935910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.935886 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:02.936226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.936203 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:02.936373 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.936229 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:02.937780 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.937757 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:02.937995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.937972 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:02.938386 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.938363 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "kube-api-access-nf4t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:02.938583 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:02.938564 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm" pod "e6338952-93f9-4634-b015-3ee549712d4d" (UID: "e6338952-93f9-4634-b015-3ee549712d4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:03.037196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.037169 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:03.037196 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.037196 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf4t6\" (UniqueName: \"kubernetes.io/projected/e6338952-93f9-4634-b015-3ee549712d4d-kube-api-access-nf4t6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:03.037374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.037205 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6338952-93f9-4634-b015-3ee549712d4d-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:03.037374 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.037213 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6338952-93f9-4634-b015-3ee549712d4d-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:03.387359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.387315 4622 generic.go:356] "Generic (PLEG): container finished" podID="e6338952-93f9-4634-b015-3ee549712d4d" containerID="064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad" exitCode=0 Oct 30 17:58:03.387359 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.387360 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" Oct 30 17:58:03.387637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.387352 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" event={"ID":"e6338952-93f9-4634-b015-3ee549712d4d","Type":"ContainerDied","Data":"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad"} Oct 30 17:58:03.387637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.387480 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d" event={"ID":"e6338952-93f9-4634-b015-3ee549712d4d","Type":"ContainerDied","Data":"a53fa13d6cd1bf8f56f2415774874c70bd5ca22550858b1ea3de08e775de1614"} Oct 30 17:58:03.387637 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.387501 4622 scope.go:122] "RemoveContainer" containerID="064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad" Oct 30 17:58:03.398831 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.398807 4622 scope.go:122] "RemoveContainer" containerID="064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad" Oct 30 17:58:03.399115 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:03.399087 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad\": container with ID starting with 064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad not found: ID does not exist" containerID="064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad" Oct 30 17:58:03.399229 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.399121 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad"} err="failed to get container status \"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad\": rpc error: code = NotFound desc = could not find container \"064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad\": container with ID starting with 064b99001991374415e5ae6d4a22847dc5f023b550d3e872636ad455b496d7ad not found: ID does not exist" Oct 30 17:58:03.439553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.439516 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:58:03.451276 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:03.451243 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-8d1062/auth-test-model-8d1062-kserve-7956ccbdb9-c2l5d"] Oct 30 17:58:04.375124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:04.375092 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6338952-93f9-4634-b015-3ee549712d4d" path="/var/lib/kubelet/pods/e6338952-93f9-4634-b015-3ee549712d4d/volumes" Oct 30 17:58:04.870148 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:04.870117 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:58:05.988500 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.988468 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 17:58:05.989103 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989086 4622 cpu_manager.go:414] "containerMap: removing container" podUID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" containerName="main" Oct 30 17:58:05.989177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989104 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" containerName="main" Oct 30 17:58:05.989177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989124 4622 cpu_manager.go:414] "containerMap: removing container" podUID="9f36a23d-9689-4b26-aeab-f22591389a76" containerName="main" Oct 30 17:58:05.989177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989130 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f36a23d-9689-4b26-aeab-f22591389a76" containerName="main" Oct 30 17:58:05.989177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989142 4622 cpu_manager.go:414] "containerMap: removing container" podUID="e6338952-93f9-4634-b015-3ee549712d4d" containerName="main" Oct 30 17:58:05.989177 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989147 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6338952-93f9-4634-b015-3ee549712d4d" containerName="main" Oct 30 17:58:05.989376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989239 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6338952-93f9-4634-b015-3ee549712d4d" containerName="main" Oct 30 17:58:05.989376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989256 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="575ef9bf-bb2a-41c8-90ea-483224bd5ef8" containerName="main" Oct 30 17:58:05.989376 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.989346 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f36a23d-9689-4b26-aeab-f22591389a76" containerName="main" Oct 30 17:58:05.994618 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:05.994595 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.014703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.014676 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fab25f-openshift-default-dockercfg-2rfnw\"" Oct 30 17:58:06.021544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.021519 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 17:58:06.096254 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096220 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmrp\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096286 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096470 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096427 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096469 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096639 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096617 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096718 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096654 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096777 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096740 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.096847 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096828 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.097018 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.096995 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199206 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199279 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199338 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199369 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199414 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199460 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199597 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199677 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmrp\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.199729 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.200580 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.201275 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.201514 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.201754 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.202285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.201984 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.211624 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.211224 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.230283 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.224563 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.249805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.249717 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.250973 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.250946 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmrp\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.319616 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.319576 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:06.644627 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:58:06.644598 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b08100e_eb64_4016_983b_64acf8052259.slice/crio-aab56034fd943a5cddf0e932630dfaeb3a007f4ad0c9db46b11adab3ae5c9e85 WatchSource:0}: Error finding container aab56034fd943a5cddf0e932630dfaeb3a007f4ad0c9db46b11adab3ae5c9e85: Status 404 returned error can't find the container with id aab56034fd943a5cddf0e932630dfaeb3a007f4ad0c9db46b11adab3ae5c9e85 Oct 30 17:58:06.647491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.647460 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:06.647581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.647525 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:06.647581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.647556 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:06.647581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:06.647557 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 17:58:07.432307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:07.430153 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerStarted","Data":"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340"} Oct 30 17:58:07.432307 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:07.430289 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerStarted","Data":"aab56034fd943a5cddf0e932630dfaeb3a007f4ad0c9db46b11adab3ae5c9e85"} Oct 30 17:58:07.490503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:07.490437 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podStartSLOduration=2.490420899 podStartE2EDuration="2.490420899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:07.486948459 +0000 UTC m=+2715.264420414" watchObservedRunningTime="2026-10-30 17:58:07.490420899 +0000 UTC m=+2715.267892861" Oct 30 17:58:08.320748 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:08.320716 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:08.327499 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:08.327475 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:08.436672 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:08.436639 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:08.437495 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:08.437478 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:58:15.840081 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.840048 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 17:58:15.844381 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.844359 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.867789 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.867766 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 17:58:15.903997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.903963 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904012 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904080 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904129 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904155 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904334 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnmv\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904427 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904413 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904501 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904476 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:15.904552 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:15.904537 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005612 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005575 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnmv\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005667 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005687 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005794 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005717 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005808 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005847 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005877 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005928 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.005971 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.005964 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.006236 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.006189 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.006324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.006233 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.006368 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.006326 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.006544 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.006505 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.006753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.006733 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.008434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.008410 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.008677 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.008646 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.028904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.028870 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.029008 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.028991 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnmv\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\") pod \"e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.159715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.159651 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:16.476538 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.476513 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 17:58:16.476956 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:58:16.476929 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2323cb_9feb_4803_92f1_8f2146ebdc43.slice/crio-cfef9526a8fdd9721cdac92dd41c5f4fd0dfb646799b49c14c7f9b4b35b9c203 WatchSource:0}: Error finding container cfef9526a8fdd9721cdac92dd41c5f4fd0dfb646799b49c14c7f9b4b35b9c203: Status 404 returned error can't find the container with id cfef9526a8fdd9721cdac92dd41c5f4fd0dfb646799b49c14c7f9b4b35b9c203 Oct 30 17:58:16.480187 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.480152 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:16.480313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.480219 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:16.480313 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.480249 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:58:16.530087 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:16.530062 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" event={"ID":"1e2323cb-9feb-4803-92f1-8f2146ebdc43","Type":"ContainerStarted","Data":"cfef9526a8fdd9721cdac92dd41c5f4fd0dfb646799b49c14c7f9b4b35b9c203"} Oct 30 17:58:17.539252 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:17.539216 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" event={"ID":"1e2323cb-9feb-4803-92f1-8f2146ebdc43","Type":"ContainerStarted","Data":"3e91242e16f43276e73773ed61eb014a2b02799edbbddbcb6d2b412e1b978510"} Oct 30 17:58:17.590469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:17.590413 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podStartSLOduration=2.590397477 podStartE2EDuration="2.590397477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:17.588981123 +0000 UTC m=+2725.366453078" watchObservedRunningTime="2026-10-30 17:58:17.590397477 +0000 UTC m=+2725.367869431" Oct 30 17:58:18.163291 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:18.162639 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:18.167044 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:18.167019 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:18.167162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:18.167068 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:19.161286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:19.160620 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:19.161286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:19.160690 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:19.892064 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:19.892015 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerName="main" containerID="cri-o://77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb" gracePeriod=45 Oct 30 17:58:19.966072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:19.966028 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.162:8000/ready\": dial tcp 10.134.0.162:8000: connect: connection refused" Oct 30 17:58:20.162336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.160645 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:20.162336 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.160704 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:20.206613 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.206591 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:58:20.286545 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286515 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.286702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286590 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.286702 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286658 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.286846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286763 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.286846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286793 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:20.286846 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286814 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.287009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286891 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\" (UniqueName: \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\") pod \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\" (UID: \"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c\") " Oct 30 17:58:20.287009 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.286938 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:20.287146 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.287120 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:20.287588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.287559 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.287730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.287592 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.287730 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.287606 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.288880 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.288853 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:20.289157 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.289138 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:20.289290 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.289209 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c" pod "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" (UID: "d1a04cf3-b810-4abb-bae6-d07abb7bfd4c"). InnerVolumeSpecName "kube-api-access-z6m9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:20.389165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.389136 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.389165 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.389164 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6m9c\" (UniqueName: \"kubernetes.io/projected/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-kube-api-access-z6m9c\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.389347 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.389176 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:58:20.582487 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.582456 4622 generic.go:356] "Generic (PLEG): container finished" podID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerID="77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb" exitCode=0 Oct 30 17:58:20.582680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.582512 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" Oct 30 17:58:20.582680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.582533 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" event={"ID":"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c","Type":"ContainerDied","Data":"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb"} Oct 30 17:58:20.582680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.582574 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54" event={"ID":"d1a04cf3-b810-4abb-bae6-d07abb7bfd4c","Type":"ContainerDied","Data":"7bdbb9eb69b93bfbea40ae4256ba91696d3883d15da13ad9f603a0596753d6e1"} Oct 30 17:58:20.582680 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.582592 4622 scope.go:122] "RemoveContainer" containerID="77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb" Oct 30 17:58:20.592690 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.592671 4622 scope.go:122] "RemoveContainer" containerID="77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb" Oct 30 17:58:20.592978 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:20.592957 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb\": container with ID starting with 77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb not found: ID does not exist" containerID="77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb" Oct 30 17:58:20.593077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.592984 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb"} err="failed to get container status \"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb\": rpc error: code = NotFound desc = could not find container \"77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb\": container with ID starting with 77db9aae5f9ac8ec46b26e433a64f50d3d9e036d72bf659e231d5f859ff293fb not found: ID does not exist" Oct 30 17:58:20.636164 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.636140 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:58:20.662001 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.661311 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-4dcdbb/auth-test-model-4dcdbb-kserve-9bff947cc-8db54"] Oct 30 17:58:20.792294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.792245 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 17:58:20.792855 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.792841 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerName="main" Oct 30 17:58:20.792924 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.792857 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerName="main" Oct 30 17:58:20.792972 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.792964 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" containerName="main" Oct 30 17:58:20.797778 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.797759 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:20.815286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.815108 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-738845-serving-cert\"" Oct 30 17:58:20.817663 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.817634 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 17:58:20.895857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.895747 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:20.895857 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.895846 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:20.896073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.895956 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwqp\" (UniqueName: \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:20.896073 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.895995 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.001320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.997694 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwqp\" (UniqueName: \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.001320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.997761 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.001320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.998010 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.001320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:20.998097 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.001320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.001304 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.011550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.002889 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.011550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.011381 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.027074 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.027044 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwqp\" (UniqueName: \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\") pod \"maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.113469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.113435 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:21.165603 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.165524 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:21.166042 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.165595 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:21.468559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.468468 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 17:58:21.472474 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:58:21.472447 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a94584_506d_4b6d_81d8_88e8ddd408d4.slice/crio-8f5bc44025106b16fc373b74f5e6cce7eba23c682168551108fd7b7fc4670d25 WatchSource:0}: Error finding container 8f5bc44025106b16fc373b74f5e6cce7eba23c682168551108fd7b7fc4670d25: Status 404 returned error can't find the container with id 8f5bc44025106b16fc373b74f5e6cce7eba23c682168551108fd7b7fc4670d25 Oct 30 17:58:21.591326 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:21.591291 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" event={"ID":"01a94584-506d-4b6d-81d8-88e8ddd408d4","Type":"ContainerStarted","Data":"8f5bc44025106b16fc373b74f5e6cce7eba23c682168551108fd7b7fc4670d25"} Oct 30 17:58:22.161122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.160640 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:22.161122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.160702 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:22.375302 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.375255 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a04cf3-b810-4abb-bae6-d07abb7bfd4c" path="/var/lib/kubelet/pods/d1a04cf3-b810-4abb-bae6-d07abb7bfd4c/volumes" Oct 30 17:58:22.609253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.609211 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" event={"ID":"01a94584-506d-4b6d-81d8-88e8ddd408d4","Type":"ContainerStarted","Data":"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07"} Oct 30 17:58:22.609429 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.609330 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:22.663294 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:22.660846 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" podStartSLOduration=2.25074599 podStartE2EDuration="2.660827434s" totalImagesPullingTime="410.081444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:22.659622069 +0000 UTC m=+2730.437094051" watchObservedRunningTime="2026-10-30 17:58:22.660827434 +0000 UTC m=+2730.438299389" Oct 30 17:58:23.160699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:23.160661 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:58:23.160878 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:23.160720 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:58:24.165653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:24.165622 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:24.166050 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:24.166029 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:24.166732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:24.166711 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 17:58:24.534746 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:24.534720 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:58:26.350284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.347810 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 17:58:26.359546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.359519 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.416133 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.415158 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 17:58:26.432979 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.416693 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-455e4a-serving-cert\"" Oct 30 17:58:26.483100 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.483059 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4f46\" (UniqueName: \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.483293 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.483190 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.483355 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.483324 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.483491 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.483475 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.584714 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.584678 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4f46\" (UniqueName: \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.584904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.584790 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.584904 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.584871 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.585032 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.585021 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.585199 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:26.585182 4622 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-serving-cert: secret "maas-api-e2e-ipp-a-455e4a-serving-cert" not found Oct 30 17:58:26.585300 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:26.585288 4622 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls podName:a615c998-26e0-49ed-8878-6b1c0f7b1d9e nodeName:}" failed. No retries permitted until 2026-10-30 17:58:27.085243365 +0000 UTC m=+2734.862715317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls") pod "maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" (UID: "a615c998-26e0-49ed-8878-6b1c0f7b1d9e") : secret "maas-api-e2e-ipp-a-455e4a-serving-cert" not found Oct 30 17:58:26.586526 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.586498 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.588320 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.588285 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:26.611358 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:26.611241 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4f46\" (UniqueName: \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:27.095063 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:27.095013 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:27.114932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:27.114895 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:27.342881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:27.341495 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:28.083286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.080961 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 17:58:28.622581 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.622553 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 17:58:28.915454 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.915414 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" event={"ID":"a615c998-26e0-49ed-8878-6b1c0f7b1d9e","Type":"ContainerStarted","Data":"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad"} Oct 30 17:58:28.915653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.915483 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" event={"ID":"a615c998-26e0-49ed-8878-6b1c0f7b1d9e","Type":"ContainerStarted","Data":"ecf27539176b7309040ebb82a881a8c3e241def2ae5efa5669a8b06a00f1d8b3"} Oct 30 17:58:28.915653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.915513 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:28.960508 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:28.960445 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" podStartSLOduration=2.6654994739999998 podStartE2EDuration="2.960428195s" totalImagesPullingTime="294.928721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:28.956842973 +0000 UTC m=+2736.734314928" watchObservedRunningTime="2026-10-30 17:58:28.960428195 +0000 UTC m=+2736.737900157" Oct 30 17:58:34.928553 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:34.928521 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 17:58:47.528366 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:47.528326 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:58:54.431507 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.431476 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 17:58:54.436986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.436969 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.464681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.464637 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fab25f-serving-cert\"" Oct 30 17:58:54.475557 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.475529 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 17:58:54.535020 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:58:54.534989 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:58:54.588856 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.588825 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.589071 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.589051 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqlz\" (UniqueName: \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.589124 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.589101 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.589183 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.589170 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.690987 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.691062 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.691132 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.691408 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqlz\" (UniqueName: \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.692801 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.706286 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.705488 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.710424 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.707170 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.727324 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.725521 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqlz\" (UniqueName: \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\") pod \"maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:54.752346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:54.750575 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:55.059312 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:55.059286 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 17:58:55.059516 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:58:55.059495 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116cf21b_2633_4a7c_aebf_3d7855282cce.slice/crio-98f0a8447cfb817017157546fc980678605dd6f1e1b563707a23acb189ffba6b WatchSource:0}: Error finding container 98f0a8447cfb817017157546fc980678605dd6f1e1b563707a23acb189ffba6b: Status 404 returned error can't find the container with id 98f0a8447cfb817017157546fc980678605dd6f1e1b563707a23acb189ffba6b Oct 30 17:58:55.163321 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:55.163285 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" event={"ID":"116cf21b-2633-4a7c-aebf-3d7855282cce","Type":"ContainerStarted","Data":"98f0a8447cfb817017157546fc980678605dd6f1e1b563707a23acb189ffba6b"} Oct 30 17:58:56.171934 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:56.171897 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" event={"ID":"116cf21b-2633-4a7c-aebf-3d7855282cce","Type":"ContainerStarted","Data":"da4d9299b0210a6f8f18546d3f1b2e30b5e3989a640f8a514a51d152bf42e161"} Oct 30 17:58:56.172467 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:56.171979 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:58:56.213445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:58:56.213394 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" podStartSLOduration=1.7934147660000002 podStartE2EDuration="2.213376506s" totalImagesPullingTime="419.96174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:58:56.213171757 +0000 UTC m=+2763.990643712" watchObservedRunningTime="2026-10-30 17:58:56.213376506 +0000 UTC m=+2763.990848458" Oct 30 17:59:02.184035 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.184003 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 17:59:02.554348 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.554313 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" podUID="0dbf3931-fd28-4a21-9f7c-f75687666150" containerName="main" containerID="cri-o://55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04" gracePeriod=45 Oct 30 17:59:02.835588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.835567 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:59:02.893516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893487 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.893686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893546 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\" (UniqueName: \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.893686 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893638 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.893824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893746 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.893929 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893899 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:02.894000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893964 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.894075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.893997 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:02.894075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.894054 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\") pod \"0dbf3931-fd28-4a21-9f7c-f75687666150\" (UID: \"0dbf3931-fd28-4a21-9f7c-f75687666150\") " Oct 30 17:59:02.894363 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.894341 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:02.894699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.894669 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:02.894699 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.894694 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:02.894858 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.894709 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:02.896010 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.895988 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:02.896186 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.896166 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:59:02.896469 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.896448 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd" pod "0dbf3931-fd28-4a21-9f7c-f75687666150" (UID: "0dbf3931-fd28-4a21-9f7c-f75687666150"). InnerVolumeSpecName "kube-api-access-c6fsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:02.996077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.996048 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fsd\" (UniqueName: \"kubernetes.io/projected/0dbf3931-fd28-4a21-9f7c-f75687666150-kube-api-access-c6fsd\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:02.996077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.996076 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dbf3931-fd28-4a21-9f7c-f75687666150-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:02.996077 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:02.996084 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dbf3931-fd28-4a21-9f7c-f75687666150-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:03.220800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.220718 4622 generic.go:356] "Generic (PLEG): container finished" podID="0dbf3931-fd28-4a21-9f7c-f75687666150" containerID="55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04" exitCode=0 Oct 30 17:59:03.220800 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.220768 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" Oct 30 17:59:03.221351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.220809 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" event={"ID":"0dbf3931-fd28-4a21-9f7c-f75687666150","Type":"ContainerDied","Data":"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04"} Oct 30 17:59:03.221351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.220857 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp" event={"ID":"0dbf3931-fd28-4a21-9f7c-f75687666150","Type":"ContainerDied","Data":"00dde96144453680be5773e62844e67c98bebb8b4ad6ce9cc693e19f060c15c8"} Oct 30 17:59:03.221351 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.220880 4622 scope.go:122] "RemoveContainer" containerID="55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04" Oct 30 17:59:03.232201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.232183 4622 scope.go:122] "RemoveContainer" containerID="55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04" Oct 30 17:59:03.232497 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:59:03.232478 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04\": container with ID starting with 55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04 not found: ID does not exist" containerID="55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04" Oct 30 17:59:03.232569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.232510 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04"} err="failed to get container status \"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04\": rpc error: code = NotFound desc = could not find container \"55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04\": container with ID starting with 55b2d3330ce10d0e730a91f7ae79fd594b2c40bca76a3543008444463ad30c04 not found: ID does not exist" Oct 30 17:59:03.274710 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.274679 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:59:03.278072 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:03.278054 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/auth-test-model-a23c98-kserve-96cd58b97-l79fp"] Oct 30 17:59:04.376384 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:04.376353 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dbf3931-fd28-4a21-9f7c-f75687666150" path="/var/lib/kubelet/pods/0dbf3931-fd28-4a21-9f7c-f75687666150/volumes" Oct 30 17:59:15.512179 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.512147 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 17:59:15.515060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.512910 4622 cpu_manager.go:414] "containerMap: removing container" podUID="0dbf3931-fd28-4a21-9f7c-f75687666150" containerName="main" Oct 30 17:59:15.515060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.512927 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbf3931-fd28-4a21-9f7c-f75687666150" containerName="main" Oct 30 17:59:15.515060 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.513013 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbf3931-fd28-4a21-9f7c-f75687666150" containerName="main" Oct 30 17:59:15.517851 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.517831 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.532995 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.532971 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 17:59:15.547640 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.547615 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"openshift-service-ca.crt\"" Oct 30 17:59:15.547967 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.547946 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"rate-test-model-a2a6cb-kserve-self-signed-certs\"" Oct 30 17:59:15.549002 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.548863 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"default-dockercfg-tsvtz\"" Oct 30 17:59:15.549135 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.548953 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a2a6cb\"/\"kube-root-ca.crt\"" Oct 30 17:59:15.704747 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.704712 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.704952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.704814 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.704952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.704867 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.704952 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.704917 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.705122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.704987 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6jd\" (UniqueName: \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.705122 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.705025 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.809066 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.809029 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.809888 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.809862 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.810413 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.810394 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.810907 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.810890 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.811437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.811416 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6jd\" (UniqueName: \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.811910 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.811891 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.812602 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.809738 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.812753 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.810324 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.812870 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.810800 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.832284 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.818176 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.843061 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.843024 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:15.847926 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:15.847897 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6jd\" (UniqueName: \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\") pod \"rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:16.133141 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:16.133050 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:16.471713 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:16.471673 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 17:59:17.402874 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:17.402831 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" event={"ID":"a219676e-3e52-4177-b0f0-1ec855b5d044","Type":"ContainerStarted","Data":"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63"} Oct 30 17:59:17.403411 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:17.402888 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" event={"ID":"a219676e-3e52-4177-b0f0-1ec855b5d044","Type":"ContainerStarted","Data":"6c4a7c34aa7c908bad24a37da0cfb9fed6040d021d4b4204c4b71c032ff2165b"} Oct 30 17:59:17.460352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:17.460251 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" podStartSLOduration=2.46023149 podStartE2EDuration="2.46023149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:17.459716495 +0000 UTC m=+2785.237188462" watchObservedRunningTime="2026-10-30 17:59:17.46023149 +0000 UTC m=+2785.237703447" Oct 30 17:59:18.134239 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:18.134203 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:19.567938 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.567873 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 17:59:19.581154 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.581099 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.603349 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.603320 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 17:59:19.609726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.609668 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-90b48a\"/\"auth-test-model-90b48a-kserve-self-signed-certs\"" Oct 30 17:59:19.609726 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.609691 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-90b48a\"/\"kube-root-ca.crt\"" Oct 30 17:59:19.609949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.609872 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-90b48a\"/\"openshift-service-ca.crt\"" Oct 30 17:59:19.637916 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.637715 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-90b48a\"/\"default-dockercfg-vdnw8\"" Oct 30 17:59:19.691939 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.691895 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.692353 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.692317 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.692546 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.692525 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.692801 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.692777 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.693074 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.693052 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjtp\" (UniqueName: \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.693303 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.693283 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795209 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795168 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795344 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795433 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795511 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795462 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795575 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795731 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795701 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjtp\" (UniqueName: \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795828 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795703 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.795997 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.795977 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.796191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.796168 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.798075 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.798034 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.798921 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.798894 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.819390 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.819333 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjtp\" (UniqueName: \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\") pod \"auth-test-model-90b48a-kserve-b4b678f5c-vjgf2\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:19.910285 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:19.909469 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:20.342342 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:59:20.342305 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod445413df_51d2_4282_bf19_ce2e6587227d.slice/crio-87de8c952a9263b712502a315710296ba21c809a07dc61bfb360350644b6d590 WatchSource:0}: Error finding container 87de8c952a9263b712502a315710296ba21c809a07dc61bfb360350644b6d590: Status 404 returned error can't find the container with id 87de8c952a9263b712502a315710296ba21c809a07dc61bfb360350644b6d590 Oct 30 17:59:20.343584 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:20.343556 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 17:59:20.434333 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:20.434298 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" event={"ID":"445413df-51d2-4282-bf19-ce2e6587227d","Type":"ContainerStarted","Data":"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5"} Oct 30 17:59:20.434463 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:20.434350 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" event={"ID":"445413df-51d2-4282-bf19-ce2e6587227d","Type":"ContainerStarted","Data":"87de8c952a9263b712502a315710296ba21c809a07dc61bfb360350644b6d590"} Oct 30 17:59:20.476204 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:20.476153 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" podStartSLOduration=1.476138569 podStartE2EDuration="1.476138569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:20.474649054 +0000 UTC m=+2788.252121011" watchObservedRunningTime="2026-10-30 17:59:20.476138569 +0000 UTC m=+2788.253610523" Oct 30 17:59:20.910587 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:20.910554 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:22.043836 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.043744 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk"] Oct 30 17:59:22.055146 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.055116 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65850e4d-w2rhk"] Oct 30 17:59:22.375516 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.375486 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f71e34a-5d72-4cfd-be6a-4b28ef395c4e" path="/var/lib/kubelet/pods/2f71e34a-5d72-4cfd-be6a-4b28ef395c4e/volumes" Oct 30 17:59:22.455519 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.455489 4622 generic.go:356] "Generic (PLEG): container finished" podID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerID="b9f5a9a05d9d4987ea8c2e4488ec968c14afa0c27eebe4b3db9cc6568e44ed88" exitCode=0 Oct 30 17:59:22.455681 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.455567 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerDied","Data":"b9f5a9a05d9d4987ea8c2e4488ec968c14afa0c27eebe4b3db9cc6568e44ed88"} Oct 30 17:59:22.455987 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.455972 4622 scope.go:122] "RemoveContainer" containerID="b9f5a9a05d9d4987ea8c2e4488ec968c14afa0c27eebe4b3db9cc6568e44ed88" Oct 30 17:59:22.457631 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.457604 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:22.457738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.457672 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:22.457738 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:22.457701 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:23.475022 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:23.474984 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerStarted","Data":"c6741dd5ef212932efd3a569e3779233b90db5c1753b6ffc8c50259dd126b993"} Oct 30 17:59:23.975030 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:23.974990 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:59:23.976178 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:23.976147 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:59:23.976330 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:23.976199 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 17:59:24.188577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:24.188545 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:59:24.597379 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:59:24.595986 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:59:24.974510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:24.974427 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:59:24.974510 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:24.974477 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 17:59:25.974257 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:25.974218 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:59:25.974694 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:25.974326 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 17:59:26.133420 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:26.133389 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:26.172661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:26.172633 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:26.568503 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:26.568472 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 17:59:26.974181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:26.974106 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:59:26.974181 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:26.974155 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 17:59:27.974949 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:27.974905 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 17:59:27.975389 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:27.974967 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 17:59:28.037773 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.037733 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:59:28.038000 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.037974 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" containerID="cri-o://a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2" gracePeriod=30 Oct 30 17:59:28.626316 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.626281 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:59:28.626540 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.626496 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" containerID="cri-o://651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446" gracePeriod=30 Oct 30 17:59:28.980661 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.980592 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:59:28.981651 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:28.981629 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 17:59:29.893569 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.893528 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 17:59:29.899159 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.899138 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.910798 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.910774 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:29.933322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.928121 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-d207f8-openshift-default-dockercfg-d2z4p\"" Oct 30 17:59:29.933322 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.933061 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:29.934842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.934816 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 17:59:29.997554 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997523 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997684 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbd2h\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997723 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997751 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997810 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997849 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997878 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.997984 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.997959 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:29.998350 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:29.998023 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099201 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099169 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbd2h\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099224 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099257 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099360 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099296 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099385 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099426 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099550 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099487 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099553 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099703 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099670 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099706 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.099974 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.099907 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.100130 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.100054 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.100174 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.100136 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.100230 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.100214 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.101824 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.101799 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.102182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.102164 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.122653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.122623 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.122752 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.122737 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbd2h\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\") pod \"e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.213401 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.213342 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:30.543148 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:59:30.543112 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de896bf_fccf_4677_9b62_97038fc12bca.slice/crio-892ea75d5da5f352d5dd614aeb285eb5444dbbee3784625c24d62be460d694e6 WatchSource:0}: Error finding container 892ea75d5da5f352d5dd614aeb285eb5444dbbee3784625c24d62be460d694e6: Status 404 returned error can't find the container with id 892ea75d5da5f352d5dd614aeb285eb5444dbbee3784625c24d62be460d694e6 Oct 30 17:59:30.545571 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.545544 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 17:59:30.545950 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.545920 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:30.546020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.546000 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:30.546076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.546031 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:30.568033 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.567999 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" event={"ID":"3de896bf-fccf-4677-9b62-97038fc12bca","Type":"ContainerStarted","Data":"892ea75d5da5f352d5dd614aeb285eb5444dbbee3784625c24d62be460d694e6"} Oct 30 17:59:30.579717 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.579404 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 17:59:30.769329 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.769288 4622 patch_prober.go:29] interesting pod/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:59:30.769536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.769345 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:59:30.827447 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.827342 4622 patch_prober.go:29] interesting pod/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:59:30.827621 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:30.827471 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:59:31.578020 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:31.577973 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" event={"ID":"3de896bf-fccf-4677-9b62-97038fc12bca","Type":"ContainerStarted","Data":"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5"} Oct 30 17:59:31.636323 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:31.636258 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podStartSLOduration=2.636244361 podStartE2EDuration="2.636244361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:31.63439449 +0000 UTC m=+2799.411866445" watchObservedRunningTime="2026-10-30 17:59:31.636244361 +0000 UTC m=+2799.413716316" Oct 30 17:59:32.032812 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.032776 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l"] Oct 30 17:59:32.036188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.036166 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c0595e6-rk22l"] Oct 30 17:59:32.213695 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.213606 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:32.215253 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.215222 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:32.215437 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.215308 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:32.376641 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:32.376605 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a98425-8801-4055-b8ef-6c49b42b4d7b" path="/var/lib/kubelet/pods/17a98425-8801-4055-b8ef-6c49b42b4d7b/volumes" Oct 30 17:59:33.215308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.214384 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:33.215308 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.214461 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:33.278805 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.278770 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:59:33.478189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478096 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478189 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478177 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478218 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478478 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478336 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478513 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478682 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478664 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478760 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478712 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478821 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478806 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.478890 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.478868 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\") pod \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\" (UID: \"7f5e6c61-d33f-4392-be49-bd7b6ed21b91\") " Oct 30 17:59:33.479772 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.479743 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.480070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.480050 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:33.480191 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.480164 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.480385 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.480366 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.481664 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.481639 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.481796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.481758 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.481796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.481774 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.481796 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.481786 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.482160 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.482137 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.482536 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.482512 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:59:33.482653 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.482630 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:33.484660 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.484389 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "kube-api-access-bt46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:33.486396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.486369 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy" pod "7f5e6c61-d33f-4392-be49-bd7b6ed21b91" (UID: "7f5e6c61-d33f-4392-be49-bd7b6ed21b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:33.583040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.583012 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.583040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.583037 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.583040 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.583046 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.583342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.583054 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.583342 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.583063 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt46x\" (UniqueName: \"kubernetes.io/projected/7f5e6c61-d33f-4392-be49-bd7b6ed21b91-kube-api-access-bt46x\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:33.597518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.597488 4622 generic.go:356] "Generic (PLEG): container finished" podID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerID="a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2" exitCode=0 Oct 30 17:59:33.597633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.597540 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" Oct 30 17:59:33.597633 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.597579 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" event={"ID":"7f5e6c61-d33f-4392-be49-bd7b6ed21b91","Type":"ContainerDied","Data":"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2"} Oct 30 17:59:33.597717 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.597640 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv" event={"ID":"7f5e6c61-d33f-4392-be49-bd7b6ed21b91","Type":"ContainerDied","Data":"9d08e198a6bb1fb0af9607679c100f368dbe523ef26cf6496fecd1637da00457"} Oct 30 17:59:33.597717 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.597671 4622 scope.go:122] "RemoveContainer" containerID="a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2" Oct 30 17:59:33.609084 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.609068 4622 scope.go:122] "RemoveContainer" containerID="a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2" Oct 30 17:59:33.609357 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:59:33.609335 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2\": container with ID starting with a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2 not found: ID does not exist" containerID="a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2" Oct 30 17:59:33.609415 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.609364 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2"} err="failed to get container status \"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2\": rpc error: code = NotFound desc = could not find container \"a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2\": container with ID starting with a23ae2f5c4a89885225b5c35d8ee45467340febf6beb813f7152a292786ed9a2 not found: ID does not exist" Oct 30 17:59:33.672318 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.672168 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:59:33.676991 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.676965 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-27567a-openshift-default-6cdbd7d7c-ls5fv"] Oct 30 17:59:33.919070 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:33.919043 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005577 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005625 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005651 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005673 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005707 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005730 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005769 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005820 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006136 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.005903 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\") pod \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\" (UID: \"d309e61b-f5f9-41c7-9ffa-0a8ba650df11\") " Oct 30 17:59:34.006853 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.006766 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:34.008715 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.008686 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:34.008894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.008868 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:34.009492 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.009398 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:34.009650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.009612 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:59:34.009650 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.009634 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:59:34.011606 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.011575 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:59:34.012131 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.012097 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:34.012226 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.012140 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7" pod "d309e61b-f5f9-41c7-9ffa-0a8ba650df11" (UID: "d309e61b-f5f9-41c7-9ffa-0a8ba650df11"). InnerVolumeSpecName "kube-api-access-xkbh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:59:34.107271 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107230 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107257 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107301 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107312 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107320 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107330 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107338 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107347 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbh7\" (UniqueName: \"kubernetes.io/projected/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-kube-api-access-xkbh7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.107402 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.107355 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d309e61b-f5f9-41c7-9ffa-0a8ba650df11-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 17:59:34.214575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.214539 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:34.214732 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.214592 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:34.402834 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.402799 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" path="/var/lib/kubelet/pods/7f5e6c61-d33f-4392-be49-bd7b6ed21b91/volumes" Oct 30 17:59:34.606989 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.606951 4622 generic.go:356] "Generic (PLEG): container finished" podID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerID="651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446" exitCode=0 Oct 30 17:59:34.607176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.607023 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" Oct 30 17:59:34.607176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.607036 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" event={"ID":"d309e61b-f5f9-41c7-9ffa-0a8ba650df11","Type":"ContainerDied","Data":"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446"} Oct 30 17:59:34.607176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.607116 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc" event={"ID":"d309e61b-f5f9-41c7-9ffa-0a8ba650df11","Type":"ContainerDied","Data":"0d48e5b341db74ae8fa0950c55d5a645eb7ef9ff33f650857e7346f55208d9dc"} Oct 30 17:59:34.607176 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.607140 4622 scope.go:122] "RemoveContainer" containerID="651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446" Oct 30 17:59:34.617881 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.617861 4622 scope.go:122] "RemoveContainer" containerID="651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446" Oct 30 17:59:34.618151 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:59:34.618128 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446\": container with ID starting with 651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446 not found: ID does not exist" containerID="651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446" Oct 30 17:59:34.618218 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.618163 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446"} err="failed to get container status \"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446\": rpc error: code = NotFound desc = could not find container \"651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446\": container with ID starting with 651c777ae1a53e35ac71019708a93c4f1b5fdacdeca5687145de5cddead08446 not found: ID does not exist" Oct 30 17:59:34.665986 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.665890 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:59:34.674788 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:34.674757 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-306aed-openshift-default-6d946b5549-8nmnc"] Oct 30 17:59:35.214403 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:35.214362 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:35.214750 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:35.214478 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:36.213839 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:36.213804 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:36.214245 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:36.213859 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:36.375768 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:36.375733 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" path="/var/lib/kubelet/pods/d309e61b-f5f9-41c7-9ffa-0a8ba650df11/volumes" Oct 30 17:59:37.214300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:37.213719 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 17:59:37.214300 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:37.213762 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 17:59:38.218783 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:38.218754 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:38.219223 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:38.219202 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:38.219951 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:38.219928 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 17:59:51.382483 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.382406 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 17:59:51.382994 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.382980 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" Oct 30 17:59:51.383048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.382995 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" Oct 30 17:59:51.383048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.383039 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" Oct 30 17:59:51.383048 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.383045 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" Oct 30 17:59:51.383162 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.383152 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5e6c61-d33f-4392-be49-bd7b6ed21b91" containerName="istio-proxy" Oct 30 17:59:51.383203 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.383168 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d309e61b-f5f9-41c7-9ffa-0a8ba650df11" containerName="istio-proxy" Oct 30 17:59:51.386911 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.386887 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.416634 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.416599 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 17:59:51.508955 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.508918 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509116 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.508978 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jst4\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509188 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509159 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509306 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509274 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509364 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509347 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509430 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509413 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509468 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509458 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509623 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509606 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.509705 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.509691 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.611687 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.611634 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612675 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612641 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612715 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612842 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612767 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jst4\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612843 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612901 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.612966 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612949 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.613123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612985 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.613123 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.613033 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.613577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.612523 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.613577 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.613522 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.613767 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.613651 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.614128 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.614099 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.614231 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.614157 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.618372 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.618325 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.618543 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.618524 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.641782 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.641717 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.642518 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.642492 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jst4\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\") pod \"e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:51.717457 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:51.717422 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:52.050894 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.050864 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 17:59:52.051936 ip-10-0-143-147 kubenswrapper[4622]: W1030 17:59:52.051896 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd1506e_d8a3_432d_8180_60377fe3c5a1.slice/crio-7d394cf3d64dbf0c340131f2b5ffdeb7c53ab3fb347c122ed049957fd524f996 WatchSource:0}: Error finding container 7d394cf3d64dbf0c340131f2b5ffdeb7c53ab3fb347c122ed049957fd524f996: Status 404 returned error can't find the container with id 7d394cf3d64dbf0c340131f2b5ffdeb7c53ab3fb347c122ed049957fd524f996 Oct 30 17:59:52.054781 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.054744 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:52.054875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.054814 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:52.054875 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.054844 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:52.757727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.757694 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" event={"ID":"7bd1506e-d8a3-432d-8180-60377fe3c5a1","Type":"ContainerStarted","Data":"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46"} Oct 30 17:59:52.758168 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.757752 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" event={"ID":"7bd1506e-d8a3-432d-8180-60377fe3c5a1","Type":"ContainerStarted","Data":"7d394cf3d64dbf0c340131f2b5ffdeb7c53ab3fb347c122ed049957fd524f996"} Oct 30 17:59:52.815391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:52.815339 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podStartSLOduration=1.8153222690000002 podStartE2EDuration="1.815322269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:59:52.814464137 +0000 UTC m=+2820.591936093" watchObservedRunningTime="2026-10-30 17:59:52.815322269 +0000 UTC m=+2820.592794224" Oct 30 17:59:53.510625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.509364 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": read tcp 10.134.0.2:58546->10.134.0.171:15021: read: connection reset by peer" start-of-body= Oct 30 17:59:53.510625 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.509440 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": read tcp 10.134.0.2:58546->10.134.0.171:15021: read: connection reset by peer" Oct 30 17:59:53.717932 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.717852 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:53.719434 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.719408 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:59:53.719573 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.719466 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 17:59:53.773352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.773258 4622 generic.go:356] "Generic (PLEG): container finished" podID="2b08100e-eb64-4016-983b-64acf8052259" containerID="8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340" exitCode=0 Oct 30 17:59:53.773352 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.773332 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerDied","Data":"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340"} Oct 30 17:59:53.773826 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.773810 4622 scope.go:122] "RemoveContainer" containerID="8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340" Oct 30 17:59:53.775445 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.775411 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:53.775575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.775498 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:53.775575 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:53.775548 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 17:59:54.535182 ip-10-0-143-147 kubenswrapper[4622]: E1030 17:59:54.535146 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 17:59:54.718391 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:54.718359 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:59:54.718486 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:54.718426 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 17:59:54.809659 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:54.809571 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerStarted","Data":"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a"} Oct 30 17:59:55.320396 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:55.320357 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 17:59:55.320588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:55.320471 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:59:55.320588 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:55.320535 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:59:55.718539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:55.718466 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:59:55.718539 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:55.718513 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 17:59:56.320832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:56.320665 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:59:56.320832 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:56.320738 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:59:56.717917 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:56.717833 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:59:56.718182 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:56.718154 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 17:59:57.320140 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:57.320096 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:59:57.320346 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:57.320153 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:59:57.718799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:57.718715 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 17:59:57.718799 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:57.718776 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 17:59:58.320559 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:58.320524 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:59:58.320727 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:58.320590 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:59:58.723213 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:58.723134 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:58.723594 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:58.723473 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:58.724076 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:58.724056 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 17:59:59.320098 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:59.320060 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:59:59.320305 ip-10-0-143-147 kubenswrapper[4622]: I1030 17:59:59.320121 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 18:00:00.270994 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.270955 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:00.278709 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.278682 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.286844 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.286818 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:00.294153 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.293445 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.304719 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.304688 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:00.319113 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.319085 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:00.328578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.328551 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:00.328827 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.328673 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.328981 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.328957 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 18:00:00.329963 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.329942 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 18:00:00.331054 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.330804 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 18:00:00.352443 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.352413 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:00.358012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.357994 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.362160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.362140 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:00.363999 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.363980 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:00:00.370627 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.367564 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.382476 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.381614 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:00.386846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.386821 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:00.392312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.392288 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:00.392859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.392838 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.399521 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.399497 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:00.399634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.399594 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.402057 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.402031 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:00.402831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.402772 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfs8\" (UniqueName: \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.403002 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.402981 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.403078 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403032 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkzh\" (UniqueName: \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.403147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403078 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzhf\" (UniqueName: \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.403209 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403185 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.403303 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403251 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.403372 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403355 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.403473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403454 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdxd\" (UniqueName: \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.403546 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403500 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qx8\" (UniqueName: \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.403546 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.403535 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.407057 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.406980 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:00.419516 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.419493 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:00.505224 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505183 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdxd\" (UniqueName: \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.505428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505231 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qx8\" (UniqueName: \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.505428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505257 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.505428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505372 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfs8\" (UniqueName: \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.505621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505449 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ns9\" (UniqueName: \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.505621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505479 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbj5\" (UniqueName: \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.505621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.505508 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.506058 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506029 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.506151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506081 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.506151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506083 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkzh\" (UniqueName: \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.506286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506238 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.506286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506248 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzhf\" (UniqueName: \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.506418 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506381 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.506480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506435 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.506525 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506492 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.506579 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506540 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.507010 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.506989 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.507108 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.507093 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.507163 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.507104 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.529352 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.529294 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdxd\" (UniqueName: \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.529485 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.529362 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qx8\" (UniqueName: \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.529485 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.529396 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfs8\" (UniqueName: \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\") pod \"maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.529686 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.529667 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkzh\" (UniqueName: \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.529753 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.529680 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzhf\" (UniqueName: \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.609186 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.609133 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ns9\" (UniqueName: \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.609186 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.609188 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbj5\" (UniqueName: \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.609499 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.609233 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.609571 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.609500 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.615289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.610575 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.615289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.610891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.615289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.615126 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:00.628880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.628841 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:00.643012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.642969 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbj5\" (UniqueName: \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\") pod \"maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.652939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.652909 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ns9\" (UniqueName: \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\") pod \"maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:00.660152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.659941 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:00.696572 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.696541 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:00.704650 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.704624 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 18:00:00.724134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.724097 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:00.731699 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.731671 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:00.731851 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:00.731778 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:01.306463 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.306425 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d2291a_7d88_4dc6_9eb1_e36c0b7dcadf.slice/crio-3b90cd82e24b46214e605bab50e1cd00517e94b80b18679aca4fe184e312d0e8 WatchSource:0}: Error finding container 3b90cd82e24b46214e605bab50e1cd00517e94b80b18679aca4fe184e312d0e8: Status 404 returned error can't find the container with id 3b90cd82e24b46214e605bab50e1cd00517e94b80b18679aca4fe184e312d0e8 Oct 30 18:00:01.307115 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.307015 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:01.320299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.320273 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:01.324953 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.324929 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c68c5c_9944_426e_9c42_6d08d737c3bf.slice/crio-4d741da821bd9a1404395387bd41f2f7f0a07d22a5b83f4952c8891809b01975 WatchSource:0}: Error finding container 4d741da821bd9a1404395387bd41f2f7f0a07d22a5b83f4952c8891809b01975: Status 404 returned error can't find the container with id 4d741da821bd9a1404395387bd41f2f7f0a07d22a5b83f4952c8891809b01975 Oct 30 18:00:01.346347 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.346321 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2bf873_ce37_4a2d_960c_d8fb7aafde23.slice/crio-53f6b21c381a5543f5a6a3691fa68b0f5627835c0ef9f8801b211cd01739a8db WatchSource:0}: Error finding container 53f6b21c381a5543f5a6a3691fa68b0f5627835c0ef9f8801b211cd01739a8db: Status 404 returned error can't find the container with id 53f6b21c381a5543f5a6a3691fa68b0f5627835c0ef9f8801b211cd01739a8db Oct 30 18:00:01.347771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.347749 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:01.397342 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.396172 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6950800_6464_4aec_bf23_39472d5bf684.slice/crio-521dd7819dfe334c3f8340831ffa6ee2e81b74d5372fa6dcb35a5f3edda27ab5 WatchSource:0}: Error finding container 521dd7819dfe334c3f8340831ffa6ee2e81b74d5372fa6dcb35a5f3edda27ab5: Status 404 returned error can't find the container with id 521dd7819dfe334c3f8340831ffa6ee2e81b74d5372fa6dcb35a5f3edda27ab5 Oct 30 18:00:01.397342 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.396614 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:01.420580 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.420546 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4ebb41_f88d_42cb_935a_ee4450d80e2e.slice/crio-b50efc04a6f05e60c61c0511e1fa3545f199fb38dc963b1ce9833050ff40a00a WatchSource:0}: Error finding container b50efc04a6f05e60c61c0511e1fa3545f199fb38dc963b1ce9833050ff40a00a: Status 404 returned error can't find the container with id b50efc04a6f05e60c61c0511e1fa3545f199fb38dc963b1ce9833050ff40a00a Oct 30 18:00:01.423955 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.423930 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:01.443201 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.443178 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:01.493830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.493802 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:01.642740 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:01.642658 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727bc4ad_ddf9_485d_91a0_ca5b1965a94e.slice/crio-b0fbb3dd72cdaa0ccd3458aa18cb0200c52db11cfb439489b030edc9b8d27393 WatchSource:0}: Error finding container b0fbb3dd72cdaa0ccd3458aa18cb0200c52db11cfb439489b030edc9b8d27393: Status 404 returned error can't find the container with id b0fbb3dd72cdaa0ccd3458aa18cb0200c52db11cfb439489b030edc9b8d27393 Oct 30 18:00:01.995853 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.995822 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerStarted","Data":"b50efc04a6f05e60c61c0511e1fa3545f199fb38dc963b1ce9833050ff40a00a"} Oct 30 18:00:01.997440 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.997410 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerStarted","Data":"40df2f88712b6b5c1a66c5897041cb9b11e364c0ff3bb256913b71b1303e3a6b"} Oct 30 18:00:01.997588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.997461 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerStarted","Data":"3b90cd82e24b46214e605bab50e1cd00517e94b80b18679aca4fe184e312d0e8"} Oct 30 18:00:01.998713 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:01.998687 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerStarted","Data":"b0fbb3dd72cdaa0ccd3458aa18cb0200c52db11cfb439489b030edc9b8d27393"} Oct 30 18:00:02.000199 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.000175 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerStarted","Data":"7ab9f92ceee20df6a4390db2782405f8744704da90f6a87243764a6595b7f554"} Oct 30 18:00:02.000320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.000226 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerStarted","Data":"53f6b21c381a5543f5a6a3691fa68b0f5627835c0ef9f8801b211cd01739a8db"} Oct 30 18:00:02.001793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.001768 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerStarted","Data":"eff54ade09de55a4a445e122582c417eb49dfa5fb93339af44a14f9d2955a548"} Oct 30 18:00:02.001889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.001813 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerStarted","Data":"4d741da821bd9a1404395387bd41f2f7f0a07d22a5b83f4952c8891809b01975"} Oct 30 18:00:02.003349 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.003325 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerStarted","Data":"586cfd31134ddb923ba8a6933ac754872a6c23d6ce9e54b4bf93cadb6cbe2a29"} Oct 30 18:00:02.003422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.003370 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerStarted","Data":"521dd7819dfe334c3f8340831ffa6ee2e81b74d5372fa6dcb35a5f3edda27ab5"} Oct 30 18:00:02.004412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.004391 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerStarted","Data":"57a090fdd29094eb6ad976ed88b2a360e43610582b4c57501a11e004e2176d8d"} Oct 30 18:00:02.037169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.037121 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" podStartSLOduration=2.037104833 podStartE2EDuration="2.037104833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.036182331 +0000 UTC m=+2829.813654287" watchObservedRunningTime="2026-10-30 18:00:02.037104833 +0000 UTC m=+2829.814576789" Oct 30 18:00:02.151518 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.151454 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" podStartSLOduration=2.151434131 podStartE2EDuration="2.151434131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.084164018 +0000 UTC m=+2829.861635974" watchObservedRunningTime="2026-10-30 18:00:02.151434131 +0000 UTC m=+2829.928906088" Oct 30 18:00:02.164845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.164781 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" podStartSLOduration=2.164759151 podStartE2EDuration="2.164759151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.148786591 +0000 UTC m=+2829.926258546" watchObservedRunningTime="2026-10-30 18:00:02.164759151 +0000 UTC m=+2829.942231106" Oct 30 18:00:02.189921 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:02.189869 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" podStartSLOduration=2.189854438 podStartE2EDuration="2.189854438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:02.187797639 +0000 UTC m=+2829.965269596" watchObservedRunningTime="2026-10-30 18:00:02.189854438 +0000 UTC m=+2829.967326392" Oct 30 18:00:03.013704 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.013608 4622 generic.go:356] "Generic (PLEG): container finished" podID="f6950800-6464-4aec-bf23-39472d5bf684" containerID="586cfd31134ddb923ba8a6933ac754872a6c23d6ce9e54b4bf93cadb6cbe2a29" exitCode=60 Oct 30 18:00:03.014155 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.013696 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerDied","Data":"586cfd31134ddb923ba8a6933ac754872a6c23d6ce9e54b4bf93cadb6cbe2a29"} Oct 30 18:00:03.014155 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.014037 4622 scope.go:122] "RemoveContainer" containerID="586cfd31134ddb923ba8a6933ac754872a6c23d6ce9e54b4bf93cadb6cbe2a29" Oct 30 18:00:03.015622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.015604 4622 generic.go:356] "Generic (PLEG): container finished" podID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerID="73c1e5eff493d3fd7d4ab7c87fbd807d16823bc75e83aba8fc35b981e623434a" exitCode=60 Oct 30 18:00:03.015723 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.015663 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerDied","Data":"73c1e5eff493d3fd7d4ab7c87fbd807d16823bc75e83aba8fc35b981e623434a"} Oct 30 18:00:03.016022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.015976 4622 scope.go:122] "RemoveContainer" containerID="73c1e5eff493d3fd7d4ab7c87fbd807d16823bc75e83aba8fc35b981e623434a" Oct 30 18:00:03.017487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.017467 4622 generic.go:356] "Generic (PLEG): container finished" podID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerID="ee4fb1d959edff4f8e1eb3e23796ec7e1e3ef78e27e56171972c46e76b49523c" exitCode=60 Oct 30 18:00:03.017577 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.017564 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerDied","Data":"ee4fb1d959edff4f8e1eb3e23796ec7e1e3ef78e27e56171972c46e76b49523c"} Oct 30 18:00:03.017904 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.017830 4622 scope.go:122] "RemoveContainer" containerID="ee4fb1d959edff4f8e1eb3e23796ec7e1e3ef78e27e56171972c46e76b49523c" Oct 30 18:00:03.020045 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.020025 4622 generic.go:356] "Generic (PLEG): container finished" podID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerID="40df2f88712b6b5c1a66c5897041cb9b11e364c0ff3bb256913b71b1303e3a6b" exitCode=60 Oct 30 18:00:03.020131 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.020106 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerDied","Data":"40df2f88712b6b5c1a66c5897041cb9b11e364c0ff3bb256913b71b1303e3a6b"} Oct 30 18:00:03.020358 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.020337 4622 scope.go:122] "RemoveContainer" containerID="40df2f88712b6b5c1a66c5897041cb9b11e364c0ff3bb256913b71b1303e3a6b" Oct 30 18:00:03.021917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.021899 4622 generic.go:356] "Generic (PLEG): container finished" podID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerID="f5d8b9c7b5efd2479e84cc9475364de1b3ab7df88e42b589dc8afcf2a49ff81e" exitCode=60 Oct 30 18:00:03.022019 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.021954 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerDied","Data":"f5d8b9c7b5efd2479e84cc9475364de1b3ab7df88e42b589dc8afcf2a49ff81e"} Oct 30 18:00:03.022206 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.022191 4622 scope.go:122] "RemoveContainer" containerID="f5d8b9c7b5efd2479e84cc9475364de1b3ab7df88e42b589dc8afcf2a49ff81e" Oct 30 18:00:03.023703 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.023682 4622 generic.go:356] "Generic (PLEG): container finished" podID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerID="7ab9f92ceee20df6a4390db2782405f8744704da90f6a87243764a6595b7f554" exitCode=60 Oct 30 18:00:03.023806 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.023758 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerDied","Data":"7ab9f92ceee20df6a4390db2782405f8744704da90f6a87243764a6595b7f554"} Oct 30 18:00:03.023941 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.023927 4622 scope.go:122] "RemoveContainer" containerID="7ab9f92ceee20df6a4390db2782405f8744704da90f6a87243764a6595b7f554" Oct 30 18:00:03.025620 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.025601 4622 generic.go:356] "Generic (PLEG): container finished" podID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerID="eff54ade09de55a4a445e122582c417eb49dfa5fb93339af44a14f9d2955a548" exitCode=60 Oct 30 18:00:03.025726 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.025630 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerDied","Data":"eff54ade09de55a4a445e122582c417eb49dfa5fb93339af44a14f9d2955a548"} Oct 30 18:00:03.025898 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:03.025880 4622 scope.go:122] "RemoveContainer" containerID="eff54ade09de55a4a445e122582c417eb49dfa5fb93339af44a14f9d2955a548" Oct 30 18:00:04.035432 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.035399 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerStarted","Data":"d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b"} Oct 30 18:00:04.037536 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.037510 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerStarted","Data":"77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07"} Oct 30 18:00:04.039997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.039974 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerStarted","Data":"26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0"} Oct 30 18:00:04.043588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.043223 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerStarted","Data":"c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a"} Oct 30 18:00:04.046041 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.046011 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerStarted","Data":"e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4"} Oct 30 18:00:04.048607 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.048583 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerStarted","Data":"acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451"} Oct 30 18:00:04.050652 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.050632 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerStarted","Data":"ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119"} Oct 30 18:00:04.076840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.076799 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" podStartSLOduration=4.076786033 podStartE2EDuration="4.076786033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.074653597 +0000 UTC m=+2831.852125551" watchObservedRunningTime="2026-10-30 18:00:04.076786033 +0000 UTC m=+2831.854257988" Oct 30 18:00:04.243782 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.243724 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" podStartSLOduration=4.243704227 podStartE2EDuration="4.243704227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.239456238 +0000 UTC m=+2832.016928194" watchObservedRunningTime="2026-10-30 18:00:04.243704227 +0000 UTC m=+2832.021176184" Oct 30 18:00:04.280359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:04.280318 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" podStartSLOduration=4.280301004 podStartE2EDuration="4.280301004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:04.278898844 +0000 UTC m=+2832.056370798" watchObservedRunningTime="2026-10-30 18:00:04.280301004 +0000 UTC m=+2832.057772962" Oct 30 18:00:05.060803 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.060771 4622 generic.go:356] "Generic (PLEG): container finished" podID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerID="d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b" exitCode=60 Oct 30 18:00:05.061251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.060858 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerDied","Data":"d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b"} Oct 30 18:00:05.061251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.060933 4622 scope.go:122] "RemoveContainer" containerID="ee4fb1d959edff4f8e1eb3e23796ec7e1e3ef78e27e56171972c46e76b49523c" Oct 30 18:00:05.061251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.061137 4622 scope.go:122] "RemoveContainer" containerID="d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b" Oct 30 18:00:05.061562 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.061542 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2_odh-ai-gateway-infra(bf4ebb41-f88d-42cb-935a-ee4450d80e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" Oct 30 18:00:05.066982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.063477 4622 generic.go:356] "Generic (PLEG): container finished" podID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerID="77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07" exitCode=60 Oct 30 18:00:05.066982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.063564 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerDied","Data":"77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07"} Oct 30 18:00:05.066982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.063939 4622 scope.go:122] "RemoveContainer" containerID="77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07" Oct 30 18:00:05.066982 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.064307 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm_odh-ai-gateway-infra(17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" Oct 30 18:00:05.067279 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.067021 4622 generic.go:356] "Generic (PLEG): container finished" podID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerID="26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0" exitCode=60 Oct 30 18:00:05.067279 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.067047 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerDied","Data":"26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0"} Oct 30 18:00:05.067409 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.067390 4622 scope.go:122] "RemoveContainer" containerID="26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0" Oct 30 18:00:05.067706 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.067677 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl_odh-ai-gateway-infra(727bc4ad-ddf9-485d-91a0-ca5b1965a94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" Oct 30 18:00:05.070488 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.070462 4622 generic.go:356] "Generic (PLEG): container finished" podID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerID="c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a" exitCode=60 Oct 30 18:00:05.070582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.070504 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerDied","Data":"c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a"} Oct 30 18:00:05.070800 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.070783 4622 scope.go:122] "RemoveContainer" containerID="c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a" Oct 30 18:00:05.071073 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.071054 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f_odh-ai-gateway-infra(4f2bf873-ce37-4a2d-960c-d8fb7aafde23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" Oct 30 18:00:05.072657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.072635 4622 generic.go:356] "Generic (PLEG): container finished" podID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerID="e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4" exitCode=60 Oct 30 18:00:05.072785 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.072763 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerDied","Data":"e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4"} Oct 30 18:00:05.072946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.072930 4622 scope.go:122] "RemoveContainer" containerID="e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4" Oct 30 18:00:05.073202 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.073184 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk_odh-ai-gateway-infra(a7c68c5c-9944-426e-9c42-6d08d737c3bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" Oct 30 18:00:05.074785 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.074764 4622 generic.go:356] "Generic (PLEG): container finished" podID="f6950800-6464-4aec-bf23-39472d5bf684" containerID="acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451" exitCode=60 Oct 30 18:00:05.074865 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.074848 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerDied","Data":"acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451"} Oct 30 18:00:05.075067 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.075050 4622 scope.go:122] "RemoveContainer" containerID="acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451" Oct 30 18:00:05.075386 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.075366 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54_odh-ai-gateway-infra(f6950800-6464-4aec-bf23-39472d5bf684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" podUID="f6950800-6464-4aec-bf23-39472d5bf684" Oct 30 18:00:05.076816 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.076796 4622 generic.go:356] "Generic (PLEG): container finished" podID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerID="ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119" exitCode=60 Oct 30 18:00:05.076931 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.076848 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerDied","Data":"ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119"} Oct 30 18:00:05.077169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.077155 4622 scope.go:122] "RemoveContainer" containerID="ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119" Oct 30 18:00:05.077432 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:05.077412 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl_odh-ai-gateway-infra(5181b4ec-a59c-403f-a6b7-1fe7e25f8f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" Oct 30 18:00:05.080762 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.080740 4622 scope.go:122] "RemoveContainer" containerID="40df2f88712b6b5c1a66c5897041cb9b11e364c0ff3bb256913b71b1303e3a6b" Oct 30 18:00:05.093726 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.093708 4622 scope.go:122] "RemoveContainer" containerID="f5d8b9c7b5efd2479e84cc9475364de1b3ab7df88e42b589dc8afcf2a49ff81e" Oct 30 18:00:05.109060 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.109042 4622 scope.go:122] "RemoveContainer" containerID="7ab9f92ceee20df6a4390db2782405f8744704da90f6a87243764a6595b7f554" Oct 30 18:00:05.122275 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.122247 4622 scope.go:122] "RemoveContainer" containerID="eff54ade09de55a4a445e122582c417eb49dfa5fb93339af44a14f9d2955a548" Oct 30 18:00:05.135784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.135766 4622 scope.go:122] "RemoveContainer" containerID="586cfd31134ddb923ba8a6933ac754872a6c23d6ce9e54b4bf93cadb6cbe2a29" Oct 30 18:00:05.148786 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:05.148768 4622 scope.go:122] "RemoveContainer" containerID="73c1e5eff493d3fd7d4ab7c87fbd807d16823bc75e83aba8fc35b981e623434a" Oct 30 18:00:06.087302 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.087253 4622 scope.go:122] "RemoveContainer" containerID="acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451" Oct 30 18:00:06.087705 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.087641 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54_odh-ai-gateway-infra(f6950800-6464-4aec-bf23-39472d5bf684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" podUID="f6950800-6464-4aec-bf23-39472d5bf684" Oct 30 18:00:06.089388 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.089365 4622 scope.go:122] "RemoveContainer" containerID="ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119" Oct 30 18:00:06.089675 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.089651 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl_odh-ai-gateway-infra(5181b4ec-a59c-403f-a6b7-1fe7e25f8f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" Oct 30 18:00:06.091061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.091041 4622 scope.go:122] "RemoveContainer" containerID="d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b" Oct 30 18:00:06.091393 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.091373 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2_odh-ai-gateway-infra(bf4ebb41-f88d-42cb-935a-ee4450d80e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" Oct 30 18:00:06.092667 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.092649 4622 scope.go:122] "RemoveContainer" containerID="77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07" Oct 30 18:00:06.092917 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.092898 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm_odh-ai-gateway-infra(17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" Oct 30 18:00:06.094154 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.094133 4622 scope.go:122] "RemoveContainer" containerID="26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0" Oct 30 18:00:06.094416 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.094396 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl_odh-ai-gateway-infra(727bc4ad-ddf9-485d-91a0-ca5b1965a94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" Oct 30 18:00:06.095955 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.095927 4622 scope.go:122] "RemoveContainer" containerID="c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a" Oct 30 18:00:06.096335 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.096307 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f_odh-ai-gateway-infra(4f2bf873-ce37-4a2d-960c-d8fb7aafde23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" Oct 30 18:00:06.097589 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:06.097570 4622 scope.go:122] "RemoveContainer" containerID="e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4" Oct 30 18:00:06.097784 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:06.097767 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk_odh-ai-gateway-infra(a7c68c5c-9944-426e-9c42-6d08d737c3bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" Oct 30 18:00:11.790305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.790249 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:00:11.796328 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.796301 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.822695 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.820866 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:00:11.846614 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.846581 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-584a12-openshift-default-dockercfg-drhpj\"" Oct 30 18:00:11.870352 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870325 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870492 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870384 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870526 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870644 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870569 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rwb\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870644 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870596 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870644 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870627 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870722 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870770 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.870825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.870811 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973228 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973317 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973438 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973482 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rwb\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973506 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973535 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973569 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973612 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.976286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.973654 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.993013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.992979 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.993516 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.993491 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.993991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.993970 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.994375 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.994355 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:11.996732 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:11.996710 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.006504 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.006479 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.034285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.030818 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.039191 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.035888 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.042513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.040168 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rwb\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\") pod \"e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.116983 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.116956 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:12.466086 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:12.466041 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb792b_b985_43cc_ac13_3ce3d7832dfb.slice/crio-3eca819b7b160b6016ec2a61e460eb89a1ee0bb050761ce5011d860076e4e729 WatchSource:0}: Error finding container 3eca819b7b160b6016ec2a61e460eb89a1ee0bb050761ce5011d860076e4e729: Status 404 returned error can't find the container with id 3eca819b7b160b6016ec2a61e460eb89a1ee0bb050761ce5011d860076e4e729 Oct 30 18:00:12.468602 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.468571 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:00:12.469687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.469650 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:12.469748 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.469734 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:12.469794 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:12.469780 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:13.166017 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:13.165980 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerStarted","Data":"8726fba6b45aa836e3e86de4e407e2a44a7206d5a001517ac4c082a76d4f1044"} Oct 30 18:00:13.166449 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:13.166046 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerStarted","Data":"3eca819b7b160b6016ec2a61e460eb89a1ee0bb050761ce5011d860076e4e729"} Oct 30 18:00:14.117545 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.117507 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:14.124403 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.124373 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:14.186654 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.186605 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podStartSLOduration=3.186591193 podStartE2EDuration="3.186591193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:13.230514847 +0000 UTC m=+2841.007986803" watchObservedRunningTime="2026-10-30 18:00:14.186591193 +0000 UTC m=+2841.964063148" Oct 30 18:00:14.187919 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.187891 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:14.189484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.189456 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:00:14.367022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:14.366997 4622 scope.go:122] "RemoveContainer" containerID="d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b" Oct 30 18:00:15.196672 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:15.196642 4622 generic.go:356] "Generic (PLEG): container finished" podID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerID="5b5c4af777d0a85727b19aefd502b05cd8fc20018147818b2d6df9bd98ab0382" exitCode=60 Oct 30 18:00:15.197061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:15.196716 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerDied","Data":"5b5c4af777d0a85727b19aefd502b05cd8fc20018147818b2d6df9bd98ab0382"} Oct 30 18:00:15.197061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:15.196769 4622 scope.go:122] "RemoveContainer" containerID="d32323c9dfa0f05587be07b5b4799ffe121adc0765ca9f3465def9b1c072e31b" Oct 30 18:00:15.197344 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:15.197324 4622 scope.go:122] "RemoveContainer" containerID="5b5c4af777d0a85727b19aefd502b05cd8fc20018147818b2d6df9bd98ab0382" Oct 30 18:00:15.197654 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:15.197628 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2_odh-ai-gateway-infra(bf4ebb41-f88d-42cb-935a-ee4450d80e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" Oct 30 18:00:15.739425 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:15.739381 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerName="main" containerID="cri-o://6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d" gracePeriod=45 Oct 30 18:00:16.035826 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.035793 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 18:00:16.148743 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.148709 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\" (UniqueName: \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.148836 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.148940 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149045 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149107 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149281 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149147 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\") pod \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\" (UID: \"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e\") " Oct 30 18:00:16.149380 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149358 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:16.149526 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149495 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:16.149708 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149682 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:16.149980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149958 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.150077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149985 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.150077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.149998 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.151638 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.151612 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:16.151920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.151880 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:16.152171 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.152147 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq" pod "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" (UID: "dc23ea3f-5df7-4f79-879c-3b5b7d8d413e"). InnerVolumeSpecName "kube-api-access-6p4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:16.207308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.207283 4622 generic.go:356] "Generic (PLEG): container finished" podID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerID="6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d" exitCode=0 Oct 30 18:00:16.207649 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.207381 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" Oct 30 18:00:16.207649 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.207423 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" event={"ID":"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e","Type":"ContainerDied","Data":"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d"} Oct 30 18:00:16.207649 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.207468 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" event={"ID":"dc23ea3f-5df7-4f79-879c-3b5b7d8d413e","Type":"ContainerDied","Data":"5662d273addc87267e2f60d8afc01c9454efd1485b1834f16359bccefa337ecf"} Oct 30 18:00:16.207649 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.207487 4622 scope.go:122] "RemoveContainer" containerID="6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d" Oct 30 18:00:16.218623 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.218605 4622 scope.go:122] "RemoveContainer" containerID="6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d" Oct 30 18:00:16.218900 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:16.218879 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d\": container with ID starting with 6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d not found: ID does not exist" containerID="6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d" Oct 30 18:00:16.218977 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.218910 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d"} err="failed to get container status \"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d\": rpc error: code = NotFound desc = could not find container \"6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d\": container with ID starting with 6b6140065f1b0781cde8df56f0182a0a0e547e1294a5c67693b0fb365cea9f4d not found: ID does not exist" Oct 30 18:00:16.251369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.251132 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.251369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.251161 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.251369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.251176 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4tq\" (UniqueName: \"kubernetes.io/projected/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e-kube-api-access-6p4tq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.251545 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.251491 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:16.268079 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.265600 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 18:00:16.275417 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.275386 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd"] Oct 30 18:00:16.379588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.379552 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" path="/var/lib/kubelet/pods/dc23ea3f-5df7-4f79-879c-3b5b7d8d413e/volumes" Oct 30 18:00:16.380951 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.380926 4622 scope.go:122] "RemoveContainer" containerID="26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0" Oct 30 18:00:16.406735 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.406713 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:16.455006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.454561 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\" (UniqueName: \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\") pod \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " Oct 30 18:00:16.455006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.454634 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\") pod \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\" (UID: \"bf4ebb41-f88d-42cb-935a-ee4450d80e2e\") " Oct 30 18:00:16.458606 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.456630 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca" pod "bf4ebb41-f88d-42cb-935a-ee4450d80e2e" (UID: "bf4ebb41-f88d-42cb-935a-ee4450d80e2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:16.461408 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.461381 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh" pod "bf4ebb41-f88d-42cb-935a-ee4450d80e2e" (UID: "bf4ebb41-f88d-42cb-935a-ee4450d80e2e"). InnerVolumeSpecName "kube-api-access-ztkzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:16.557563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.557539 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztkzh\" (UniqueName: \"kubernetes.io/projected/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-kube-api-access-ztkzh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.557689 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.557566 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf4ebb41-f88d-42cb-935a-ee4450d80e2e-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:16.919786 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:16.919677 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-d53ec1/sub-test-model-d53ec1-kserve-75c9f46f79-r57cd" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.167:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:00:17.216905 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.216843 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" Oct 30 18:00:17.217312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.216837 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2" event={"ID":"bf4ebb41-f88d-42cb-935a-ee4450d80e2e","Type":"ContainerDied","Data":"b50efc04a6f05e60c61c0511e1fa3545f199fb38dc963b1ce9833050ff40a00a"} Oct 30 18:00:17.217312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.216968 4622 scope.go:122] "RemoveContainer" containerID="5b5c4af777d0a85727b19aefd502b05cd8fc20018147818b2d6df9bd98ab0382" Oct 30 18:00:17.239644 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.239615 4622 generic.go:356] "Generic (PLEG): container finished" podID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerID="25cd04de44abd2389280a6fa7a5d741259dbfd85e0fe1f6e8bf01d2310f0aed5" exitCode=60 Oct 30 18:00:17.239872 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.239849 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerDied","Data":"25cd04de44abd2389280a6fa7a5d741259dbfd85e0fe1f6e8bf01d2310f0aed5"} Oct 30 18:00:17.240092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.240040 4622 scope.go:122] "RemoveContainer" containerID="26ef52fb6e53359101fd2e146d8c03da24bd175bff893a88d3ee09a81b84a4a0" Oct 30 18:00:17.240512 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.240483 4622 scope.go:122] "RemoveContainer" containerID="25cd04de44abd2389280a6fa7a5d741259dbfd85e0fe1f6e8bf01d2310f0aed5" Oct 30 18:00:17.240971 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:17.240949 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl_odh-ai-gateway-infra(727bc4ad-ddf9-485d-91a0-ca5b1965a94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" Oct 30 18:00:17.279349 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.279328 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:17.285523 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.285501 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-8d1062-29889720-4j2z2"] Oct 30 18:00:17.367275 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:17.367246 4622 scope.go:122] "RemoveContainer" containerID="ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119" Oct 30 18:00:18.259813 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.259720 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerStarted","Data":"39e7a14131d638b2ba15ec54bc0fc8097067052e99aad4d02a296777638df7c2"} Oct 30 18:00:18.346176 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.346144 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:18.388480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.380875 4622 scope.go:122] "RemoveContainer" containerID="e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4" Oct 30 18:00:18.388480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.382598 4622 scope.go:122] "RemoveContainer" containerID="77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07" Oct 30 18:00:18.402957 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.400155 4622 scope.go:122] "RemoveContainer" containerID="acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451" Oct 30 18:00:18.421395 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.421362 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" path="/var/lib/kubelet/pods/bf4ebb41-f88d-42cb-935a-ee4450d80e2e/volumes" Oct 30 18:00:18.585022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.585000 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:18.726643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.726609 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\") pod \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " Oct 30 18:00:18.726830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.726810 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\" (UniqueName: \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\") pod \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\" (UID: \"727bc4ad-ddf9-485d-91a0-ca5b1965a94e\") " Oct 30 18:00:18.727049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.727021 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca" pod "727bc4ad-ddf9-485d-91a0-ca5b1965a94e" (UID: "727bc4ad-ddf9-485d-91a0-ca5b1965a94e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:18.727336 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.727316 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:18.742991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.742964 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8" pod "727bc4ad-ddf9-485d-91a0-ca5b1965a94e" (UID: "727bc4ad-ddf9-485d-91a0-ca5b1965a94e"). InnerVolumeSpecName "kube-api-access-l5qx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:18.829372 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:18.829312 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qx8\" (UniqueName: \"kubernetes.io/projected/727bc4ad-ddf9-485d-91a0-ca5b1965a94e-kube-api-access-l5qx8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:19.272677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.272599 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerStarted","Data":"845fd1e84a9521441b446a8cd4632865547e9ae4cd2219e203f7ee302eae2923"} Oct 30 18:00:19.274887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.274861 4622 generic.go:356] "Generic (PLEG): container finished" podID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerID="39e7a14131d638b2ba15ec54bc0fc8097067052e99aad4d02a296777638df7c2" exitCode=60 Oct 30 18:00:19.275025 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.274936 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerDied","Data":"39e7a14131d638b2ba15ec54bc0fc8097067052e99aad4d02a296777638df7c2"} Oct 30 18:00:19.275025 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.274994 4622 scope.go:122] "RemoveContainer" containerID="ea3c13b5aeac4ebd16d0ce7ff7e567904eed4ba7558ab69c804ee1590a91f119" Oct 30 18:00:19.275324 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.275307 4622 scope.go:122] "RemoveContainer" containerID="39e7a14131d638b2ba15ec54bc0fc8097067052e99aad4d02a296777638df7c2" Oct 30 18:00:19.275628 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:19.275602 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl_odh-ai-gateway-infra(5181b4ec-a59c-403f-a6b7-1fe7e25f8f31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" Oct 30 18:00:19.277034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.277013 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerStarted","Data":"67e34ab362449a6699bc6e9518c792ec2eb262d5a3f85815b95c767e99b5e7ee"} Oct 30 18:00:19.278393 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.278376 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" Oct 30 18:00:19.278464 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.278375 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl" event={"ID":"727bc4ad-ddf9-485d-91a0-ca5b1965a94e","Type":"ContainerDied","Data":"b0fbb3dd72cdaa0ccd3458aa18cb0200c52db11cfb439489b030edc9b8d27393"} Oct 30 18:00:19.280459 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.280428 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerStarted","Data":"ebd298a8f86239f9e76b0ecc54276e8b67ccba76b4406dbd6820fb712d72c7f8"} Oct 30 18:00:19.288933 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.288917 4622 scope.go:122] "RemoveContainer" containerID="25cd04de44abd2389280a6fa7a5d741259dbfd85e0fe1f6e8bf01d2310f0aed5" Oct 30 18:00:19.310548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.310524 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:00:19.311243 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311224 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311245 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311304 4622 cpu_manager.go:414] "containerMap: removing container" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311314 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311337 4622 cpu_manager.go:414] "containerMap: removing container" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311344 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311359 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311366 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311382 4622 cpu_manager.go:414] "containerMap: removing container" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311389 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311405 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerName="main" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311412 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerName="main" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311435 4622 cpu_manager.go:414] "containerMap: removing container" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311443 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311610 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311635 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311646 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311654 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311663 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" containerName="cleanup" Oct 30 18:00:19.311724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.311674 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc23ea3f-5df7-4f79-879c-3b5b7d8d413e" containerName="main" Oct 30 18:00:19.317243 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.317218 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:19.317398 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.317383 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.333701 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.333677 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-738845\"/\"default-dockercfg-t75r4\"" Oct 30 18:00:19.336164 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.336137 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:00:19.336995 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.336896 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-738845\"/\"kube-root-ca.crt\"" Oct 30 18:00:19.336995 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.336953 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-738845\"/\"auth-test-model-738845-kserve-self-signed-certs\"" Oct 30 18:00:19.336995 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.336972 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-738845\"/\"openshift-service-ca.crt\"" Oct 30 18:00:19.367714 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.367688 4622 scope.go:122] "RemoveContainer" containerID="c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a" Oct 30 18:00:19.436011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.435980 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.436237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.436030 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.436237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.436084 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwqh\" (UniqueName: \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.437593 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.437051 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.437593 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.437094 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.437593 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.437130 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.441114 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.441087 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:19.452507 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.452476 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-d207f8-29889720-cksbl"] Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.539753 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.539800 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.539832 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.539958 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.540023 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.540288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.540064 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwqh\" (UniqueName: \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.556256 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.549602 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.556256 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.549865 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.556256 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.550099 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.642287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.638629 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwqh\" (UniqueName: \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.646151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.643002 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.646151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.643236 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\") pod \"auth-test-model-738845-kserve-66f6dcc97c-lmrzb\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:19.646151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:19.643682 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:20.051117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.051083 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:00:20.051241 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:20.051146 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod465185c0_eef3_44d5_9b65_ec78a387a45d.slice/crio-d9958dc256837c17100922af852a40d249b101955f0f792023e8e22e29d13e74 WatchSource:0}: Error finding container d9958dc256837c17100922af852a40d249b101955f0f792023e8e22e29d13e74: Status 404 returned error can't find the container with id d9958dc256837c17100922af852a40d249b101955f0f792023e8e22e29d13e74 Oct 30 18:00:20.289752 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.289705 4622 generic.go:356] "Generic (PLEG): container finished" podID="f6950800-6464-4aec-bf23-39472d5bf684" containerID="845fd1e84a9521441b446a8cd4632865547e9ae4cd2219e203f7ee302eae2923" exitCode=60 Oct 30 18:00:20.290167 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.289775 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerDied","Data":"845fd1e84a9521441b446a8cd4632865547e9ae4cd2219e203f7ee302eae2923"} Oct 30 18:00:20.290167 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.289892 4622 scope.go:122] "RemoveContainer" containerID="acbd458e5c01ce64587f75ee25857afe1a37ca58bd40759682f1beb4c3d28451" Oct 30 18:00:20.290291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.290245 4622 scope.go:122] "RemoveContainer" containerID="845fd1e84a9521441b446a8cd4632865547e9ae4cd2219e203f7ee302eae2923" Oct 30 18:00:20.290615 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:20.290588 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54_odh-ai-gateway-infra(f6950800-6464-4aec-bf23-39472d5bf684)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" podUID="f6950800-6464-4aec-bf23-39472d5bf684" Oct 30 18:00:20.293372 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.293351 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" event={"ID":"465185c0-eef3-44d5-9b65-ec78a387a45d","Type":"ContainerStarted","Data":"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0"} Oct 30 18:00:20.293450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.293380 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" event={"ID":"465185c0-eef3-44d5-9b65-ec78a387a45d","Type":"ContainerStarted","Data":"d9958dc256837c17100922af852a40d249b101955f0f792023e8e22e29d13e74"} Oct 30 18:00:20.295707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.295690 4622 generic.go:356] "Generic (PLEG): container finished" podID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerID="67e34ab362449a6699bc6e9518c792ec2eb262d5a3f85815b95c767e99b5e7ee" exitCode=60 Oct 30 18:00:20.295819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.295771 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerDied","Data":"67e34ab362449a6699bc6e9518c792ec2eb262d5a3f85815b95c767e99b5e7ee"} Oct 30 18:00:20.296182 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.296163 4622 scope.go:122] "RemoveContainer" containerID="67e34ab362449a6699bc6e9518c792ec2eb262d5a3f85815b95c767e99b5e7ee" Oct 30 18:00:20.296531 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:20.296505 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm_odh-ai-gateway-infra(17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" Oct 30 18:00:20.299138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.299116 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerStarted","Data":"ca252f7ef05dc1c6c2373249130ab0df9398f2390dba4e04f737ffc258ce86cc"} Oct 30 18:00:20.301340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.301317 4622 generic.go:356] "Generic (PLEG): container finished" podID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerID="ebd298a8f86239f9e76b0ecc54276e8b67ccba76b4406dbd6820fb712d72c7f8" exitCode=60 Oct 30 18:00:20.301435 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.301389 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerDied","Data":"ebd298a8f86239f9e76b0ecc54276e8b67ccba76b4406dbd6820fb712d72c7f8"} Oct 30 18:00:20.301747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.301728 4622 scope.go:122] "RemoveContainer" containerID="ebd298a8f86239f9e76b0ecc54276e8b67ccba76b4406dbd6820fb712d72c7f8" Oct 30 18:00:20.302012 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:20.301991 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk_odh-ai-gateway-infra(a7c68c5c-9944-426e-9c42-6d08d737c3bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" Oct 30 18:00:20.305226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.305205 4622 scope.go:122] "RemoveContainer" containerID="77926cd3b17d4ce8543b876bc0d7f66215fafa24ba92a1113b8f9e862026cd07" Oct 30 18:00:20.319396 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.319374 4622 scope.go:122] "RemoveContainer" containerID="e9b7a37a4058badd6e8ca61e62716652c2765043c4c67de5d0038afdca989fc4" Oct 30 18:00:20.329920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.329891 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:20.383367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.382921 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727bc4ad-ddf9-485d-91a0-ca5b1965a94e" path="/var/lib/kubelet/pods/727bc4ad-ddf9-485d-91a0-ca5b1965a94e/volumes" Oct 30 18:00:20.383512 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.383443 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:20.437519 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.437471 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" podStartSLOduration=1.437456036 podStartE2EDuration="1.437456036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:20.432419679 +0000 UTC m=+2848.209891636" watchObservedRunningTime="2026-10-30 18:00:20.437456036 +0000 UTC m=+2848.214927990" Oct 30 18:00:20.463462 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.463435 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:20.501850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.501820 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:20.582526 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.582490 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\") pod \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " Oct 30 18:00:20.583017 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.582998 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\" (UniqueName: \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\") pod \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\" (UID: \"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31\") " Oct 30 18:00:20.583797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.583751 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca" pod "5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" (UID: "5181b4ec-a59c-403f-a6b7-1fe7e25f8f31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:20.586369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.586343 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf" pod "5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" (UID: "5181b4ec-a59c-403f-a6b7-1fe7e25f8f31"). InnerVolumeSpecName "kube-api-access-rkzhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:20.644190 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.644113 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:20.685289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.685252 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzhf\" (UniqueName: \"kubernetes.io/projected/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-kube-api-access-rkzhf\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:20.685289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:20.685287 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.312720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.312692 4622 generic.go:356] "Generic (PLEG): container finished" podID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerID="ca252f7ef05dc1c6c2373249130ab0df9398f2390dba4e04f737ffc258ce86cc" exitCode=60 Oct 30 18:00:21.313065 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.312762 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerDied","Data":"ca252f7ef05dc1c6c2373249130ab0df9398f2390dba4e04f737ffc258ce86cc"} Oct 30 18:00:21.313065 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.312814 4622 scope.go:122] "RemoveContainer" containerID="c592d88ebddb957483980c89f12b318b957673af57f2b2c9f0541e926861a28a" Oct 30 18:00:21.313194 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.313153 4622 scope.go:122] "RemoveContainer" containerID="ca252f7ef05dc1c6c2373249130ab0df9398f2390dba4e04f737ffc258ce86cc" Oct 30 18:00:21.313539 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:21.313510 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f_odh-ai-gateway-infra(4f2bf873-ce37-4a2d-960c-d8fb7aafde23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" Oct 30 18:00:21.318289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.318229 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" Oct 30 18:00:21.318413 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.318284 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl" event={"ID":"5181b4ec-a59c-403f-a6b7-1fe7e25f8f31","Type":"ContainerDied","Data":"57a090fdd29094eb6ad976ed88b2a360e43610582b4c57501a11e004e2176d8d"} Oct 30 18:00:21.330636 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.330590 4622 scope.go:122] "RemoveContainer" containerID="39e7a14131d638b2ba15ec54bc0fc8097067052e99aad4d02a296777638df7c2" Oct 30 18:00:21.440307 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.440016 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:21.452151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.452120 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d53ec1-29889720-5ktgl"] Oct 30 18:00:21.495329 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.495301 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:21.567697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.567629 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:21.569602 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.569574 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:21.595992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.595844 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:21.611305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.602937 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\" (UniqueName: \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\") pod \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " Oct 30 18:00:21.611305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.602985 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\") pod \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\" (UID: \"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf\") " Oct 30 18:00:21.611305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.604211 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca" pod "17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" (UID: "17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:21.611305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.606904 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5" pod "17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" (UID: "17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf"). InnerVolumeSpecName "kube-api-access-plbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:21.705309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.705256 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\") pod \"f6950800-6464-4aec-bf23-39472d5bf684\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " Oct 30 18:00:21.705511 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.705424 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\") pod \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " Oct 30 18:00:21.705511 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.705479 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\" (UniqueName: \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\") pod \"f6950800-6464-4aec-bf23-39472d5bf684\" (UID: \"f6950800-6464-4aec-bf23-39472d5bf684\") " Oct 30 18:00:21.705686 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.705656 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca" pod "f6950800-6464-4aec-bf23-39472d5bf684" (UID: "f6950800-6464-4aec-bf23-39472d5bf684"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:21.705686 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.705676 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\" (UniqueName: \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\") pod \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\" (UID: \"a7c68c5c-9944-426e-9c42-6d08d737c3bf\") " Oct 30 18:00:21.706073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.706050 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca" pod "a7c68c5c-9944-426e-9c42-6d08d737c3bf" (UID: "a7c68c5c-9944-426e-9c42-6d08d737c3bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:21.706335 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.706311 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plbj5\" (UniqueName: \"kubernetes.io/projected/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-kube-api-access-plbj5\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.706422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.706337 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.706422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.706353 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6950800-6464-4aec-bf23-39472d5bf684-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.706422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.706368 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68c5c-9944-426e-9c42-6d08d737c3bf-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.707935 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.707904 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd" pod "f6950800-6464-4aec-bf23-39472d5bf684" (UID: "f6950800-6464-4aec-bf23-39472d5bf684"). InnerVolumeSpecName "kube-api-access-zjdxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:21.708601 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.708578 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9" pod "a7c68c5c-9944-426e-9c42-6d08d737c3bf" (UID: "a7c68c5c-9944-426e-9c42-6d08d737c3bf"). InnerVolumeSpecName "kube-api-access-25ns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:21.807336 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.807302 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25ns9\" (UniqueName: \"kubernetes.io/projected/a7c68c5c-9944-426e-9c42-6d08d737c3bf-kube-api-access-25ns9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:21.807336 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:21.807332 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdxd\" (UniqueName: \"kubernetes.io/projected/f6950800-6464-4aec-bf23-39472d5bf684-kube-api-access-zjdxd\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:22.326675 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.326637 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" event={"ID":"f6950800-6464-4aec-bf23-39472d5bf684","Type":"ContainerDied","Data":"521dd7819dfe334c3f8340831ffa6ee2e81b74d5372fa6dcb35a5f3edda27ab5"} Oct 30 18:00:22.327124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.326648 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54" Oct 30 18:00:22.327124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.326741 4622 scope.go:122] "RemoveContainer" containerID="845fd1e84a9521441b446a8cd4632865547e9ae4cd2219e203f7ee302eae2923" Oct 30 18:00:22.335252 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.335229 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" Oct 30 18:00:22.335252 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.335249 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm" event={"ID":"17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf","Type":"ContainerDied","Data":"3b90cd82e24b46214e605bab50e1cd00517e94b80b18679aca4fe184e312d0e8"} Oct 30 18:00:22.339802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.339780 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" Oct 30 18:00:22.339933 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.339801 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk" event={"ID":"a7c68c5c-9944-426e-9c42-6d08d737c3bf","Type":"ContainerDied","Data":"4d741da821bd9a1404395387bd41f2f7f0a07d22a5b83f4952c8891809b01975"} Oct 30 18:00:22.340716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.340696 4622 scope.go:122] "RemoveContainer" containerID="67e34ab362449a6699bc6e9518c792ec2eb262d5a3f85815b95c767e99b5e7ee" Oct 30 18:00:22.354110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.354090 4622 scope.go:122] "RemoveContainer" containerID="ebd298a8f86239f9e76b0ecc54276e8b67ccba76b4406dbd6820fb712d72c7f8" Oct 30 18:00:22.378187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.378157 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" path="/var/lib/kubelet/pods/5181b4ec-a59c-403f-a6b7-1fe7e25f8f31/volumes" Oct 30 18:00:22.422308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.419300 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:22.425380 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.425248 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a17354-29889720-kxg54"] Oct 30 18:00:22.491086 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.491056 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:22.494637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.494611 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-fab25f-29889720-krmzm"] Oct 30 18:00:22.505628 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.505607 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:22.528274 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.528234 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:22.533818 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.533793 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-21870f-29889720-88jbk"] Oct 30 18:00:22.619061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.618978 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\") pod \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " Oct 30 18:00:22.619231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.619173 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\" (UniqueName: \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\") pod \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\" (UID: \"4f2bf873-ce37-4a2d-960c-d8fb7aafde23\") " Oct 30 18:00:22.619437 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.619410 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca" pod "4f2bf873-ce37-4a2d-960c-d8fb7aafde23" (UID: "4f2bf873-ce37-4a2d-960c-d8fb7aafde23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:00:22.619841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.619815 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:22.621867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.621835 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8" pod "4f2bf873-ce37-4a2d-960c-d8fb7aafde23" (UID: "4f2bf873-ce37-4a2d-960c-d8fb7aafde23"). InnerVolumeSpecName "kube-api-access-ttfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:22.721235 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:22.721203 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttfs8\" (UniqueName: \"kubernetes.io/projected/4f2bf873-ce37-4a2d-960c-d8fb7aafde23-kube-api-access-ttfs8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:23.349579 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:23.349547 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" Oct 30 18:00:23.350041 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:23.349546 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f" event={"ID":"4f2bf873-ce37-4a2d-960c-d8fb7aafde23","Type":"ContainerDied","Data":"53f6b21c381a5543f5a6a3691fa68b0f5627835c0ef9f8801b211cd01739a8db"} Oct 30 18:00:23.350041 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:23.349691 4622 scope.go:122] "RemoveContainer" containerID="ca252f7ef05dc1c6c2373249130ab0df9398f2390dba4e04f737ffc258ce86cc" Oct 30 18:00:23.447067 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:23.447034 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:23.459038 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:23.459008 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-db98b0-29889720-89k9f"] Oct 30 18:00:24.374721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.374687 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" path="/var/lib/kubelet/pods/17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf/volumes" Oct 30 18:00:24.375337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.375316 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" path="/var/lib/kubelet/pods/4f2bf873-ce37-4a2d-960c-d8fb7aafde23/volumes" Oct 30 18:00:24.375986 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.375962 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" path="/var/lib/kubelet/pods/a7c68c5c-9944-426e-9c42-6d08d737c3bf/volumes" Oct 30 18:00:24.376830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.376812 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6950800-6464-4aec-bf23-39472d5bf684" path="/var/lib/kubelet/pods/f6950800-6464-4aec-bf23-39472d5bf684/volumes" Oct 30 18:00:24.536175 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:24.536138 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:00:24.806702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.806670 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:00:24.807326 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807306 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807327 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807340 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807347 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807361 4622 cpu_manager.go:414] "containerMap: removing container" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807369 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807379 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807384 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807396 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807402 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807410 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.807420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807415 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807444 4622 cpu_manager.go:414] "containerMap: removing container" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807449 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807457 4622 cpu_manager.go:414] "containerMap: removing container" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807462 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807475 4622 cpu_manager.go:414] "containerMap: removing container" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807481 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807489 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807495 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807507 4622 cpu_manager.go:414] "containerMap: removing container" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807514 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807659 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807672 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807683 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807693 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807702 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="5181b4ec-a59c-403f-a6b7-1fe7e25f8f31" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807713 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807771 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807804 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4ebb41-f88d-42cb-935a-ee4450d80e2e" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807813 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807819 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807827 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807833 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807839 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.808027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.807846 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.812888 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.812865 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.832982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.832439 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:00:24.833400 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.833370 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"kube-root-ca.crt\"" Oct 30 18:00:24.833400 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.833388 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"openshift-service-ca.crt\"" Oct 30 18:00:24.839023 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.839001 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"rate-test-model-a23c98-kserve-self-signed-certs\"" Oct 30 18:00:24.846504 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.846481 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a23c98\"/\"default-dockercfg-kjgn4\"" Oct 30 18:00:24.850138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850112 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.850234 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850169 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.850234 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850218 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.850358 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850296 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.850416 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850403 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.850474 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.850458 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjgk\" (UniqueName: \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.900567 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.900530 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.901947 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.901971 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6950800-6464-4aec-bf23-39472d5bf684" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902001 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902006 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902093 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902101 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c68c5c-9944-426e-9c42-6d08d737c3bf" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902119 4622 cpu_manager.go:414] "containerMap: removing container" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902127 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902439 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d2291a-7d88-4dc6-9eb1-e36c0b7dcadf" containerName="cleanup" Oct 30 18:00:24.903320 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.902470 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2bf873-ce37-4a2d-960c-d8fb7aafde23" containerName="cleanup" Oct 30 18:00:24.909062 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.909025 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:00:24.909209 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.909149 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.934144 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.934108 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b5cbb4-openshift-default-dockercfg-jq7qx\"" Oct 30 18:00:24.952377 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.952345 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.952779 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.952754 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953376 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953321 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953517 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953364 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953624 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953572 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953624 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953613 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953745 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953724 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.953808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953750 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.953808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953780 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.953911 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953819 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.953980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.953951 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjgk\" (UniqueName: \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.954043 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954006 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.954100 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954041 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.954100 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954074 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.954212 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954101 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdd69\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.954212 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954151 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:24.954421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954402 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.954627 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.954609 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.956145 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.956121 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.956717 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.956690 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:24.978565 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:24.978540 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjgk\" (UniqueName: \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\") pod \"rate-test-model-a23c98-kserve-7779d7b797-j2xpg\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:25.055984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.055951 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.055996 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056034 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056234 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056199 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056250 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056295 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056431 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056322 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdd69\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056431 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056374 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056531 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056492 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056531 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056522 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056815 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056832 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.056877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056863 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.057117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.056891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.058893 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.058862 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.058980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.058891 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.080544 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.080519 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdd69\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.080653 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.080516 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\") pod \"e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.131327 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.129575 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:25.243285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.242540 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:25.592659 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:00:25.592629 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod441e1089_1905_44f1_abcb_db48709d112a.slice/crio-1cd49dcd433054b5aab1891ef6b53999bbcde2960f10326eeebc002b5fc3b082 WatchSource:0}: Error finding container 1cd49dcd433054b5aab1891ef6b53999bbcde2960f10326eeebc002b5fc3b082: Status 404 returned error can't find the container with id 1cd49dcd433054b5aab1891ef6b53999bbcde2960f10326eeebc002b5fc3b082 Oct 30 18:00:25.593860 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.593832 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:00:25.668013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.667969 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:00:25.671661 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.671628 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:25.671756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.671695 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:25.671756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:25.671725 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:00:26.400341 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.400307 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" event={"ID":"441e1089-1905-44f1-abcb-db48709d112a","Type":"ContainerStarted","Data":"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c"} Oct 30 18:00:26.400517 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.400353 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" event={"ID":"441e1089-1905-44f1-abcb-db48709d112a","Type":"ContainerStarted","Data":"1cd49dcd433054b5aab1891ef6b53999bbcde2960f10326eeebc002b5fc3b082"} Oct 30 18:00:26.402160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.402133 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerStarted","Data":"eabd4a05acbbf90f781eea3a861e4533a05a501b675b4d1af02393ae6365b619"} Oct 30 18:00:26.402282 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.402180 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerStarted","Data":"03deed6b13ae84f7f967709de83cb5863b2ebeec522c63428a28e29152ec3abd"} Oct 30 18:00:26.445465 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.445416 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" podStartSLOduration=2.44540284 podStartE2EDuration="2.44540284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:26.44174709 +0000 UTC m=+2854.219219045" watchObservedRunningTime="2026-10-30 18:00:26.44540284 +0000 UTC m=+2854.222874796" Oct 30 18:00:26.487403 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:26.487357 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podStartSLOduration=2.487341996 podStartE2EDuration="2.487341996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:26.486454661 +0000 UTC m=+2854.263926616" watchObservedRunningTime="2026-10-30 18:00:26.487341996 +0000 UTC m=+2854.264813952" Oct 30 18:00:27.131047 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:27.131010 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:27.243337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:27.243303 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:27.249505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:27.249479 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:27.411751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:27.411670 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:27.412824 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:27.412803 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:00:29.644056 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:29.644022 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:29.657020 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:29.656995 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:30.484225 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:30.484196 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:00:35.130818 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:35.130785 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:35.143896 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:35.143867 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:35.628051 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:35.628020 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:00:39.994357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:39.994316 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 18:00:54.535147 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:54.535113 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:00:54.725045 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:54.725015 4622 scope.go:122] "RemoveContainer" containerID="fd3455fa209e69369b870dbaab8670a657bfd55adb8fc059c2fd44e9137bfd58" Oct 30 18:00:54.737972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:54.737949 4622 scope.go:122] "RemoveContainer" containerID="6696d97136b6aeecf5033b07ba4a0588aac384f8b82888cd30d083b060dd5754" Oct 30 18:00:54.747675 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:54.747649 4622 scope.go:122] "RemoveContainer" containerID="35510214c2c44c77f2d105389f8cd6948643bfb45af4156264d0e069f1de39c5" Oct 30 18:00:55.020257 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.020204 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerName="main" containerID="cri-o://361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0" gracePeriod=45 Oct 30 18:00:55.076902 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.076869 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": dial tcp 10.134.0.169:8000: connect: connection refused" Oct 30 18:00:55.286570 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.286542 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 18:00:55.475085 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475004 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\" (UniqueName: \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475257 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475103 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475257 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475174 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475333 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475377 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475452 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\") pod \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\" (UID: \"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe\") " Oct 30 18:00:55.475541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475454 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:55.475681 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475657 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:55.475774 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.475748 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:55.476087 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.476064 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.476087 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.476088 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.476273 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.476108 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.477563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.477540 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:00:55.477704 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.477683 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "kube-api-access-6x9z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:00:55.477819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.477746 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs" pod "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" (UID: "41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:00:55.577147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.577112 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9z4\" (UniqueName: \"kubernetes.io/projected/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-kube-api-access-6x9z4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.577147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.577135 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.577147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.577144 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:00:55.777152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.777079 4622 generic.go:356] "Generic (PLEG): container finished" podID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerID="361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0" exitCode=0 Oct 30 18:00:55.777152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.777131 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" Oct 30 18:00:55.777421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.777165 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" event={"ID":"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe","Type":"ContainerDied","Data":"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0"} Oct 30 18:00:55.777421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.777209 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4" event={"ID":"41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe","Type":"ContainerDied","Data":"0d0fdf02f41f83333df862cd8f425aae5bbbba6d85da193b915ddb68e65a0301"} Oct 30 18:00:55.777421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.777237 4622 scope.go:122] "RemoveContainer" containerID="361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0" Oct 30 18:00:55.823658 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.823633 4622 scope.go:122] "RemoveContainer" containerID="361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0" Oct 30 18:00:55.824024 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:00:55.823984 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0\": container with ID starting with 361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0 not found: ID does not exist" containerID="361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0" Oct 30 18:00:55.824225 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.824028 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0"} err="failed to get container status \"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0\": rpc error: code = NotFound desc = could not find container \"361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0\": container with ID starting with 361a4b0ceda7ff21f9d0510830083b9eeb32172f525455dbc470969829949cb0 not found: ID does not exist" Oct 30 18:00:55.849983 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.849946 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 18:00:55.856102 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:55.856065 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/sub-test-model-d207f8-kserve-54ffcb9559-4rdf4"] Oct 30 18:00:56.379858 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:00:56.379825 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" path="/var/lib/kubelet/pods/41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe/volumes" Oct 30 18:01:06.013638 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.013606 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:01:06.014196 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.014181 4622 cpu_manager.go:414] "containerMap: removing container" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerName="main" Oct 30 18:01:06.014241 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.014196 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerName="main" Oct 30 18:01:06.014392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.014380 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b331a9-ceb3-4e1d-9748-0ee1fcacb6fe" containerName="main" Oct 30 18:01:06.019531 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.019512 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.048552 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.048526 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-584a12-serving-cert\"" Oct 30 18:01:06.054541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.054508 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:01:06.114001 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.113972 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgn2\" (UniqueName: \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.114134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.114034 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.114134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.114115 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.114239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.114219 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.215498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.215473 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgn2\" (UniqueName: \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.215645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.215535 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.215645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.215582 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.215645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.215639 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.216242 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.216216 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.218109 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.218086 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.218236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.218126 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.238772 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.238753 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgn2\" (UniqueName: \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\") pod \"maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.338343 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.338302 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:06.646527 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.646496 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:01:06.646865 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:01:06.646845 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830c8d6a_6b64_4ec5_8eea_fce1f067a883.slice/crio-75b0388632ddea19613363ec10c34ca40508f0969cccdddc28ad627b1f094dd9 WatchSource:0}: Error finding container 75b0388632ddea19613363ec10c34ca40508f0969cccdddc28ad627b1f094dd9: Status 404 returned error can't find the container with id 75b0388632ddea19613363ec10c34ca40508f0969cccdddc28ad627b1f094dd9 Oct 30 18:01:06.648660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.648640 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:01:06.880521 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:06.880477 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" event={"ID":"830c8d6a-6b64-4ec5-8eea-fce1f067a883","Type":"ContainerStarted","Data":"75b0388632ddea19613363ec10c34ca40508f0969cccdddc28ad627b1f094dd9"} Oct 30 18:01:07.889195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:07.889157 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" event={"ID":"830c8d6a-6b64-4ec5-8eea-fce1f067a883","Type":"ContainerStarted","Data":"82f4b68097424e114202048c693766124db2da6b247ad44beb6de22ced7289a9"} Oct 30 18:01:07.889712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:07.889276 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:07.933270 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:07.933203 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" podStartSLOduration=2.5496928519999997 podStartE2EDuration="2.933187598s" totalImagesPullingTime="383.494746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:07.930727313 +0000 UTC m=+2895.708199268" watchObservedRunningTime="2026-10-30 18:01:07.933187598 +0000 UTC m=+2895.710659556" Oct 30 18:01:09.923145 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.923106 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:01:09.928149 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.928124 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:09.958859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.958810 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"openshift-service-ca.crt\"" Oct 30 18:01:09.958859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.958863 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"default-dockercfg-6qvqm\"" Oct 30 18:01:09.959089 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.958863 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"kube-root-ca.crt\"" Oct 30 18:01:09.959089 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.958958 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-23c1e8\"/\"rate-test-model-23c1e8-kserve-self-signed-certs\"" Oct 30 18:01:09.962929 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:09.962907 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:01:10.088605 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.088573 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.088794 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.088625 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrn6\" (UniqueName: \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.088939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.088911 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.088997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.088978 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.089068 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.089054 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.089129 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.089114 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195342 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195430 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195501 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195566 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195660 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.195829 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrn6\" (UniqueName: \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.196808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.196712 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.200282 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.196971 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.200282 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.197651 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.230507 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.230450 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.230959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.230936 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.233465 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.233442 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrn6\" (UniqueName: \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\") pod \"rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.249833 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.249555 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:10.678970 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:01:10.678938 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aec6088_65d8_406f_a00f_3c51adfc400a.slice/crio-5a3697a4de31f8bc6c051f3494db909429d3626d34d9ca2a47e5251961b51bc1 WatchSource:0}: Error finding container 5a3697a4de31f8bc6c051f3494db909429d3626d34d9ca2a47e5251961b51bc1: Status 404 returned error can't find the container with id 5a3697a4de31f8bc6c051f3494db909429d3626d34d9ca2a47e5251961b51bc1 Oct 30 18:01:10.679190 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.679170 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:01:10.693275 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.692599 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:01:10.711430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.711398 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.725745 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.724760 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:01:10.728520 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.728497 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f10dda-openshift-default-dockercfg-xvcww\"" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813228 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813414 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813465 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813504 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813567 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813616 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813736 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzlk\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813805 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.820285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.813917 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915506 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915605 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915728 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915774 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915809 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915844 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915871 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.915966 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzlk\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.916237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.916018 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.925287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.922678 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.925287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.922985 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.925287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.923225 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.946337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.925893 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.946337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.926562 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.946337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.933741 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.946337 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.935176 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.966298 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.963197 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" event={"ID":"1aec6088-65d8-406f-a00f-3c51adfc400a","Type":"ContainerStarted","Data":"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9"} Oct 30 18:01:10.966298 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.963254 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" event={"ID":"1aec6088-65d8-406f-a00f-3c51adfc400a","Type":"ContainerStarted","Data":"5a3697a4de31f8bc6c051f3494db909429d3626d34d9ca2a47e5251961b51bc1"} Oct 30 18:01:10.970297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.969035 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:10.979572 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:10.977682 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzlk\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\") pod \"e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:11.013595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.011123 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" podStartSLOduration=2.011103433 podStartE2EDuration="2.011103433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:11.010302986 +0000 UTC m=+2898.787774943" watchObservedRunningTime="2026-10-30 18:01:11.011103433 +0000 UTC m=+2898.788575392" Oct 30 18:01:11.047289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.042184 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:11.250704 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.250609 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:11.372463 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.372430 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:01:11.372463 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:01:11.372448 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1677362_ae19_4863_9ddb_0ea0a6841a16.slice/crio-bf89ef12c6b7936e06adc807a9d12e019adfcd427381133f1106493037ed488f WatchSource:0}: Error finding container bf89ef12c6b7936e06adc807a9d12e019adfcd427381133f1106493037ed488f: Status 404 returned error can't find the container with id bf89ef12c6b7936e06adc807a9d12e019adfcd427381133f1106493037ed488f Oct 30 18:01:11.375439 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.375404 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:01:11.375580 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.375466 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:01:11.375580 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.375496 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:01:11.974072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.974033 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerStarted","Data":"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b"} Oct 30 18:01:11.974566 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:11.974148 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerStarted","Data":"bf89ef12c6b7936e06adc807a9d12e019adfcd427381133f1106493037ed488f"} Oct 30 18:01:12.022158 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:12.022113 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podStartSLOduration=2.022099304 podStartE2EDuration="2.022099304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:12.018606557 +0000 UTC m=+2899.796078512" watchObservedRunningTime="2026-10-30 18:01:12.022099304 +0000 UTC m=+2899.799571259" Oct 30 18:01:12.047984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:12.047955 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:12.055069 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:12.055045 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:12.983232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:12.983193 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:12.984169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:12.984147 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:01:13.902659 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:13.902629 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:01:20.250827 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:20.250785 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:20.263198 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:20.263175 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:21.413907 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:21.413815 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:01:24.534425 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:01:24.534392 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:01:29.949641 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.949606 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:01:29.956241 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.956214 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:29.975826 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.975793 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:01:29.981232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.981186 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"default-dockercfg-lskph\"" Oct 30 18:01:29.982026 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.981310 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"rate-test-model-d207f8-kserve-self-signed-certs\"" Oct 30 18:01:29.987058 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.986940 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"kube-root-ca.crt\"" Oct 30 18:01:29.987058 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:29.987029 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-d207f8\"/\"openshift-service-ca.crt\"" Oct 30 18:01:30.023530 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023502 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.023681 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023545 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.023681 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023647 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.023812 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023787 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8qt\" (UniqueName: \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.023966 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023944 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.024005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.023998 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.125875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.125838 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126086 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.125895 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126086 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126000 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126086 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126049 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126355 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126257 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126369 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126620 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126409 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126620 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126526 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8qt\" (UniqueName: \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.126620 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.126593 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.128384 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.128362 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.129117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.129091 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.150363 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.150330 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8qt\" (UniqueName: \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\") pod \"rate-test-model-d207f8-kserve-7c89f587df-tlhnt\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.278065 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.277901 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:30.671246 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:30.671210 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:01:31.174106 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:31.174069 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" event={"ID":"59a85f43-178e-4a63-9712-30ff98dd8d43","Type":"ContainerStarted","Data":"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3"} Oct 30 18:01:31.174524 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:31.174121 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" event={"ID":"59a85f43-178e-4a63-9712-30ff98dd8d43","Type":"ContainerStarted","Data":"81940b76a811afabe7f50108042c09cbf261545fa1ce28b8ef2b5083822e33a2"} Oct 30 18:01:31.218022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:31.217965 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" podStartSLOduration=2.217948669 podStartE2EDuration="2.217948669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:31.216940054 +0000 UTC m=+2918.994412009" watchObservedRunningTime="2026-10-30 18:01:31.217948669 +0000 UTC m=+2918.995420625" Oct 30 18:01:31.278119 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:31.278086 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:40.278550 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:40.278508 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:40.291652 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:40.291621 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:41.258815 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:01:41.258776 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:01:54.542883 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:01:54.542840 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:02:02.257887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:02.257842 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 18:02:05.596808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.596775 4622 generic.go:356] "Generic (PLEG): container finished" podID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerID="8726fba6b45aa836e3e86de4e407e2a44a7206d5a001517ac4c082a76d4f1044" exitCode=0 Oct 30 18:02:05.597343 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.596852 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerDied","Data":"8726fba6b45aa836e3e86de4e407e2a44a7206d5a001517ac4c082a76d4f1044"} Oct 30 18:02:05.597343 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.597242 4622 scope.go:122] "RemoveContainer" containerID="8726fba6b45aa836e3e86de4e407e2a44a7206d5a001517ac4c082a76d4f1044" Oct 30 18:02:05.599188 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.599151 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:05.599315 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.599226 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:05.599315 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:05.599273 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:06.605899 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.605861 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerStarted","Data":"2328794aea90130b4dc484873ffae9b702bae7eebde70f5d9310994907b74027"} Oct 30 18:02:06.607710 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.607684 4622 generic.go:356] "Generic (PLEG): container finished" podID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerID="eabd4a05acbbf90f781eea3a861e4533a05a501b675b4d1af02393ae6365b619" exitCode=0 Oct 30 18:02:06.607813 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.607751 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerDied","Data":"eabd4a05acbbf90f781eea3a861e4533a05a501b675b4d1af02393ae6365b619"} Oct 30 18:02:06.608140 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.608126 4622 scope.go:122] "RemoveContainer" containerID="eabd4a05acbbf90f781eea3a861e4533a05a501b675b4d1af02393ae6365b619" Oct 30 18:02:06.609984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.609955 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:06.610059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.610026 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:06.610103 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:06.610059 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:07.117374 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:07.117342 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:02:07.117610 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:07.117586 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:07.117702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:07.117642 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:07.634314 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:07.633099 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerStarted","Data":"59d297ad68bd4537622ef7cbb67a7bd40f21efa961e2f1e639b80675fa594c8f"} Oct 30 18:02:08.117630 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:08.117591 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:08.117931 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:08.117904 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:08.250294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:08.247242 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:02:08.250294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:08.247444 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:08.250294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:08.247490 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:09.117532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:09.117487 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:09.117932 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:09.117552 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:09.248288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:09.244579 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:09.248288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:09.244650 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:10.117856 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:10.117818 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:10.118242 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:10.117875 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:10.248287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:10.244563 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:10.248287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:10.244625 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:11.117234 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:11.117199 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:11.117453 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:11.117285 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:11.266294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:11.258719 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:11.266294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:11.258828 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:12.118073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:12.118039 4622 patch_prober.go:29] interesting pod/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 18:02:12.118246 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:12.118086 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 18:02:12.246283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:12.245575 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:12.246283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:12.245633 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:12.412416 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:12.412315 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:02:13.123114 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:13.123082 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:02:13.123422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:13.123396 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:02:13.124035 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:13.124010 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:02:13.243831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:13.243792 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 18:02:13.244036 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:13.243859 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 18:02:14.247903 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:14.247874 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:02:14.248962 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:14.248937 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:02:17.295549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.295512 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerName="main" containerID="cri-o://6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63" gracePeriod=45 Oct 30 18:02:17.583284 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.583246 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 18:02:17.644973 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.644941 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\" (UniqueName: \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645143 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.644995 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645176 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645300 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645227 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\" (UniqueName: \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645374 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645336 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:17.645374 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645366 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645482 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645396 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\") pod \"a219676e-3e52-4177-b0f0-1ec855b5d044\" (UID: \"a219676e-3e52-4177-b0f0-1ec855b5d044\") " Oct 30 18:02:17.645482 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645452 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:17.645727 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.645690 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:17.646069 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.646045 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.646193 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.646074 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.646193 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.646090 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.647947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.647920 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "kube-api-access-dn6jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:17.648164 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.648137 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:17.648374 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.648351 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm" pod "a219676e-3e52-4177-b0f0-1ec855b5d044" (UID: "a219676e-3e52-4177-b0f0-1ec855b5d044"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:17.747475 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.747442 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6jd\" (UniqueName: \"kubernetes.io/projected/a219676e-3e52-4177-b0f0-1ec855b5d044-kube-api-access-dn6jd\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.747475 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.747469 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a219676e-3e52-4177-b0f0-1ec855b5d044-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.747475 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.747479 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a219676e-3e52-4177-b0f0-1ec855b5d044-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:17.805076 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.804993 4622 generic.go:356] "Generic (PLEG): container finished" podID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerID="6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63" exitCode=0 Oct 30 18:02:17.805076 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.805034 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" Oct 30 18:02:17.805309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.805073 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" event={"ID":"a219676e-3e52-4177-b0f0-1ec855b5d044","Type":"ContainerDied","Data":"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63"} Oct 30 18:02:17.805309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.805132 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" event={"ID":"a219676e-3e52-4177-b0f0-1ec855b5d044","Type":"ContainerDied","Data":"6c4a7c34aa7c908bad24a37da0cfb9fed6040d021d4b4204c4b71c032ff2165b"} Oct 30 18:02:17.805309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.805158 4622 scope.go:122] "RemoveContainer" containerID="6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63" Oct 30 18:02:17.817771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.817749 4622 scope.go:122] "RemoveContainer" containerID="6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63" Oct 30 18:02:17.818035 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:02:17.818014 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63\": container with ID starting with 6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63 not found: ID does not exist" containerID="6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63" Oct 30 18:02:17.818116 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.818049 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63"} err="failed to get container status \"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63\": rpc error: code = NotFound desc = could not find container \"6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63\": container with ID starting with 6a908d85988c58be216412205f296b4aada428570395d1861ee726e9bd0b6f63 not found: ID does not exist" Oct 30 18:02:17.858552 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.858522 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 18:02:17.862612 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:17.862592 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl"] Oct 30 18:02:18.376753 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:18.376715 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" path="/var/lib/kubelet/pods/a219676e-3e52-4177-b0f0-1ec855b5d044/volumes" Oct 30 18:02:18.534509 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:18.534474 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-a2a6cb/rate-test-model-a2a6cb-kserve-6f7d4bdfb5-qwhrl" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerName="main" probeResult="failure" output="Get \"https://10.134.0.176:8000/ready\": context deadline exceeded" Oct 30 18:02:24.534958 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:02:24.534923 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:02:27.341972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:27.341934 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:02:42.366780 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.366729 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" podUID="441e1089-1905-44f1-abcb-db48709d112a" containerName="main" containerID="cri-o://7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c" gracePeriod=45 Oct 30 18:02:42.417471 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:02:42.417440 4622 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod441e1089_1905_44f1_abcb_db48709d112a.slice/crio-conmon-7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:02:42.682346 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.682317 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:02:42.800823 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.800782 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\" (UniqueName: \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801030 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.800881 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801030 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.800902 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\" (UniqueName: \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801030 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.800947 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801054 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801119 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\") pod \"441e1089-1905-44f1-abcb-db48709d112a\" (UID: \"441e1089-1905-44f1-abcb-db48709d112a\") " Oct 30 18:02:42.801368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801201 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:42.801430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801381 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:42.801501 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801476 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:42.801819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801803 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:42.801880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801821 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:42.801880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.801831 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:42.803048 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.803027 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "kube-api-access-grjgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:42.803563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.803544 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:42.803643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.803565 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs" pod "441e1089-1905-44f1-abcb-db48709d112a" (UID: "441e1089-1905-44f1-abcb-db48709d112a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:42.912133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.903602 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjgk\" (UniqueName: \"kubernetes.io/projected/441e1089-1905-44f1-abcb-db48709d112a-kube-api-access-grjgk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:42.912133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.903643 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/441e1089-1905-44f1-abcb-db48709d112a-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:42.912133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:42.903662 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/441e1089-1905-44f1-abcb-db48709d112a-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:02:43.040008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.039955 4622 generic.go:356] "Generic (PLEG): container finished" podID="441e1089-1905-44f1-abcb-db48709d112a" containerID="7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c" exitCode=0 Oct 30 18:02:43.040232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.040058 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" Oct 30 18:02:43.040232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.040118 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" event={"ID":"441e1089-1905-44f1-abcb-db48709d112a","Type":"ContainerDied","Data":"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c"} Oct 30 18:02:43.040232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.040167 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" event={"ID":"441e1089-1905-44f1-abcb-db48709d112a","Type":"ContainerDied","Data":"1cd49dcd433054b5aab1891ef6b53999bbcde2960f10326eeebc002b5fc3b082"} Oct 30 18:02:43.040232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.040188 4622 scope.go:122] "RemoveContainer" containerID="7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c" Oct 30 18:02:43.079428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.079407 4622 scope.go:122] "RemoveContainer" containerID="7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c" Oct 30 18:02:43.079740 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:02:43.079718 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c\": container with ID starting with 7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c not found: ID does not exist" containerID="7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c" Oct 30 18:02:43.079807 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.079749 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c"} err="failed to get container status \"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c\": rpc error: code = NotFound desc = could not find container \"7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c\": container with ID starting with 7a3d513e2164445f1ea179a6dd903ac0a65792b76473b5da89f04abade3ca48c not found: ID does not exist" Oct 30 18:02:43.118806 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.118771 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:02:43.132141 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.132109 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg"] Oct 30 18:02:43.617174 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:43.617127 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-a23c98/rate-test-model-a23c98-kserve-7779d7b797-j2xpg" podUID="441e1089-1905-44f1-abcb-db48709d112a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.189:8000/ready\": context deadline exceeded" Oct 30 18:02:44.374903 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:44.374869 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441e1089-1905-44f1-abcb-db48709d112a" path="/var/lib/kubelet/pods/441e1089-1905-44f1-abcb-db48709d112a/volumes" Oct 30 18:02:52.140332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.140300 4622 generic.go:356] "Generic (PLEG): container finished" podID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerID="450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b" exitCode=0 Oct 30 18:02:52.140759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.140346 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerDied","Data":"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b"} Oct 30 18:02:52.140759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.140722 4622 scope.go:122] "RemoveContainer" containerID="450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b" Oct 30 18:02:52.142608 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.142577 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:52.142687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.142651 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:52.142687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:52.142682 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:02:53.163287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:53.158659 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerStarted","Data":"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2"} Oct 30 18:02:54.053284 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:54.048323 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:02:54.065285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:54.060557 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:02:54.065285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:54.060632 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:02:54.534738 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:02:54.534699 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:02:55.065994 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:55.065963 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:02:55.066518 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:55.066478 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:02:56.072769 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:56.072727 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:02:56.073411 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:56.073372 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:02:57.050608 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:57.050571 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:02:57.050929 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:57.050897 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:02:57.983516 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:57.983483 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:02:58.057606 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:58.057569 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 18:02:58.057872 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:58.057845 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 18:02:59.053352 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:59.053322 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:02:59.054394 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:02:59.054370 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:03:06.967890 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:06.967800 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 18:03:11.996546 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.996513 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:03:11.997182 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997166 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerName="main" Oct 30 18:03:11.997232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997184 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerName="main" Oct 30 18:03:11.997232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997194 4622 cpu_manager.go:414] "containerMap: removing container" podUID="441e1089-1905-44f1-abcb-db48709d112a" containerName="main" Oct 30 18:03:11.997232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997199 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="441e1089-1905-44f1-abcb-db48709d112a" containerName="main" Oct 30 18:03:11.997357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997309 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="441e1089-1905-44f1-abcb-db48709d112a" containerName="main" Oct 30 18:03:11.997357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.997329 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a219676e-3e52-4177-b0f0-1ec855b5d044" containerName="main" Oct 30 18:03:11.999676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:11.999654 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.023008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.022977 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-96c514-openshift-default-dockercfg-whhfb\"" Oct 30 18:03:12.058289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.045334 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064115 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064172 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064248 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064307 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064413 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzsdm\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064450 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064524 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064621 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.069721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.064668 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.165992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.165955 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzsdm\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166005 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166082 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166198 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166380 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166306 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166436 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166425 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166495 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166459 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166557 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166539 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166569 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166577 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166734 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166857 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166988 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166896 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.166988 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.166948 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.168794 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.168773 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.169149 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.169125 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.228299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.228204 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzsdm\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.240898 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.240866 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\") pod \"e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.350846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.350806 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:12.778958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.778907 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:03:12.779130 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.779052 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:03:12.779197 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.779148 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:03:12.786677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:12.786649 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:03:13.489415 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:13.489375 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerStarted","Data":"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2"} Oct 30 18:03:13.489904 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:13.489432 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerStarted","Data":"1ca778d7a67f22590ba59e1eb027fe015ec4f6d5823e6f6666e7ceeba5555d49"} Oct 30 18:03:13.553181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:13.553124 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podStartSLOduration=2.553109679 podStartE2EDuration="2.553109679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:13.548383387 +0000 UTC m=+3021.325855339" watchObservedRunningTime="2026-10-30 18:03:13.553109679 +0000 UTC m=+3021.330581634" Oct 30 18:03:14.351244 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:14.351211 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:14.358082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:14.358055 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:14.496048 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:14.496015 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:14.496996 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:14.496971 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:03:17.302109 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:17.301002 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:03:22.028807 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.028759 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" podUID="445413df-51d2-4282-bf19-ce2e6587227d" containerName="main" containerID="cri-o://e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5" gracePeriod=45 Oct 30 18:03:22.301032 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.301005 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 18:03:22.461693 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461663 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.461849 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461710 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.461895 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461844 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\" (UniqueName: \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.461936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461889 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:22.461973 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461934 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.461973 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.461963 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.462049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462016 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:22.462084 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462068 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\" (UniqueName: \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\") pod \"445413df-51d2-4282-bf19-ce2e6587227d\" (UID: \"445413df-51d2-4282-bf19-ce2e6587227d\") " Oct 30 18:03:22.462295 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462244 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:22.462675 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462655 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.462797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462679 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.462797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.462692 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.464131 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.464091 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:22.464226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.464162 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:22.464226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.464181 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp" pod "445413df-51d2-4282-bf19-ce2e6587227d" (UID: "445413df-51d2-4282-bf19-ce2e6587227d"). InnerVolumeSpecName "kube-api-access-7pjtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:22.556723 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.556693 4622 generic.go:356] "Generic (PLEG): container finished" podID="445413df-51d2-4282-bf19-ce2e6587227d" containerID="e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5" exitCode=0 Oct 30 18:03:22.556892 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.556738 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" event={"ID":"445413df-51d2-4282-bf19-ce2e6587227d","Type":"ContainerDied","Data":"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5"} Oct 30 18:03:22.556892 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.556748 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" Oct 30 18:03:22.556892 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.556764 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2" event={"ID":"445413df-51d2-4282-bf19-ce2e6587227d","Type":"ContainerDied","Data":"87de8c952a9263b712502a315710296ba21c809a07dc61bfb360350644b6d590"} Oct 30 18:03:22.556892 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.556787 4622 scope.go:122] "RemoveContainer" containerID="e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5" Oct 30 18:03:22.564558 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.564532 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pjtp\" (UniqueName: \"kubernetes.io/projected/445413df-51d2-4282-bf19-ce2e6587227d-kube-api-access-7pjtp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.564657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.564564 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/445413df-51d2-4282-bf19-ce2e6587227d-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.564657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.564579 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/445413df-51d2-4282-bf19-ce2e6587227d-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:22.571685 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.571663 4622 scope.go:122] "RemoveContainer" containerID="e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5" Oct 30 18:03:22.571967 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:22.571946 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5\": container with ID starting with e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5 not found: ID does not exist" containerID="e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5" Oct 30 18:03:22.572045 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.571974 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5"} err="failed to get container status \"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5\": rpc error: code = NotFound desc = could not find container \"e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5\": container with ID starting with e57a5b3a0cdf3afd7f6d77627e39690062fff189ab3c6cbc07589097ae7311c5 not found: ID does not exist" Oct 30 18:03:22.613304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.613256 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 18:03:22.617207 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:22.617186 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-90b48a/auth-test-model-90b48a-kserve-b4b678f5c-vjgf2"] Oct 30 18:03:24.375607 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:24.375577 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445413df-51d2-4282-bf19-ce2e6587227d" path="/var/lib/kubelet/pods/445413df-51d2-4282-bf19-ce2e6587227d/volumes" Oct 30 18:03:24.534411 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:24.534378 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:03:28.474345 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.474311 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:03:28.475229 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.475206 4622 cpu_manager.go:414] "containerMap: removing container" podUID="445413df-51d2-4282-bf19-ce2e6587227d" containerName="main" Oct 30 18:03:28.475422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.475407 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445413df-51d2-4282-bf19-ce2e6587227d" containerName="main" Oct 30 18:03:28.475747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.475732 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="445413df-51d2-4282-bf19-ce2e6587227d" containerName="main" Oct 30 18:03:28.479383 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.479364 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.498125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.498100 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-95f55d-serving-cert\"" Oct 30 18:03:28.500585 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.500562 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:03:28.547994 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.547961 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.548141 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.548037 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.548141 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.548076 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfbgv\" (UniqueName: \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.548299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.548246 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.652138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.650371 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.652138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.650947 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfbgv\" (UniqueName: \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.652138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.651103 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.654532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.653352 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.654532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.653750 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.654532 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:28.653930 4622 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-serving-cert: secret "maas-api-e2e-ipp-a-95f55d-serving-cert" not found Oct 30 18:03:28.654532 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:28.653989 4622 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls podName:f792563d-e2b2-418a-9eb7-b12dd474bf51 nodeName:}" failed. No retries permitted until 2026-10-30 18:03:29.153969858 +0000 UTC m=+3036.931441807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls") pod "maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" (UID: "f792563d-e2b2-418a-9eb7-b12dd474bf51") : secret "maas-api-e2e-ipp-a-95f55d-serving-cert" not found Oct 30 18:03:28.686934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.682726 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:28.708338 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:28.708063 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfbgv\" (UniqueName: \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:29.159946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:29.159911 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:29.162444 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:29.162418 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:29.392333 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:29.392258 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:29.848316 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:29.839153 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:03:30.621513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:30.621475 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" event={"ID":"f792563d-e2b2-418a-9eb7-b12dd474bf51","Type":"ContainerStarted","Data":"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a"} Oct 30 18:03:30.621696 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:30.621540 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" event={"ID":"f792563d-e2b2-418a-9eb7-b12dd474bf51","Type":"ContainerStarted","Data":"d335b474a74c355a134c6c72d8939bf326e613bd5383a12e4d030eab64de7889"} Oct 30 18:03:30.621696 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:30.621568 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:30.674801 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:30.674746 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" podStartSLOduration=2.299283591 podStartE2EDuration="2.674732231s" totalImagesPullingTime="375.44864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:03:30.670894745 +0000 UTC m=+3038.448366711" watchObservedRunningTime="2026-10-30 18:03:30.674732231 +0000 UTC m=+3038.452204252" Oct 30 18:03:32.327360 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.327288 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerName="main" containerID="cri-o://81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0" gracePeriod=45 Oct 30 18:03:32.450595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.450552 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.188:8000/ready\": dial tcp 10.134.0.188:8000: connect: connection refused" Oct 30 18:03:32.634814 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.634767 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:03:32.638398 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.638363 4622 generic.go:356] "Generic (PLEG): container finished" podID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerID="81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0" exitCode=0 Oct 30 18:03:32.638398 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.638397 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" Oct 30 18:03:32.638634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.638447 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" event={"ID":"465185c0-eef3-44d5-9b65-ec78a387a45d","Type":"ContainerDied","Data":"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0"} Oct 30 18:03:32.638634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.638508 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb" event={"ID":"465185c0-eef3-44d5-9b65-ec78a387a45d","Type":"ContainerDied","Data":"d9958dc256837c17100922af852a40d249b101955f0f792023e8e22e29d13e74"} Oct 30 18:03:32.638634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.638524 4622 scope.go:122] "RemoveContainer" containerID="81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0" Oct 30 18:03:32.649321 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.649271 4622 scope.go:122] "RemoveContainer" containerID="81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0" Oct 30 18:03:32.649584 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:32.649560 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0\": container with ID starting with 81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0 not found: ID does not exist" containerID="81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0" Oct 30 18:03:32.649700 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.649593 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0"} err="failed to get container status \"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0\": rpc error: code = NotFound desc = could not find container \"81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0\": container with ID starting with 81d4e2274cac32d6ee6fd4f7be8c3c5889ed95c2eb8178447d331b07fa4b39f0 not found: ID does not exist" Oct 30 18:03:32.752063 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752033 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752080 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\" (UniqueName: \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752156 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752470 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752320 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\" (UniqueName: \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752470 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752379 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752470 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752322 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:32.752470 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752459 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\") pod \"465185c0-eef3-44d5-9b65-ec78a387a45d\" (UID: \"465185c0-eef3-44d5-9b65-ec78a387a45d\") " Oct 30 18:03:32.752708 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752684 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:32.752809 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.752781 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:32.753230 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.753207 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:32.753323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.753240 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:32.753323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.753255 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:32.754563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.754537 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:03:32.754733 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.754706 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:03:32.754733 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.754721 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh" pod "465185c0-eef3-44d5-9b65-ec78a387a45d" (UID: "465185c0-eef3-44d5-9b65-ec78a387a45d"). InnerVolumeSpecName "kube-api-access-mkwqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:03:32.854796 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.854721 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/465185c0-eef3-44d5-9b65-ec78a387a45d-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:32.854796 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.854751 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/465185c0-eef3-44d5-9b65-ec78a387a45d-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:32.854796 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:32.854763 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkwqh\" (UniqueName: \"kubernetes.io/projected/465185c0-eef3-44d5-9b65-ec78a387a45d-kube-api-access-mkwqh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:03:33.026461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:33.026425 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:03:33.040687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:33.040656 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-738845/auth-test-model-738845-kserve-66f6dcc97c-lmrzb"] Oct 30 18:03:34.376392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:34.376358 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" path="/var/lib/kubelet/pods/465185c0-eef3-44d5-9b65-ec78a387a45d/volumes" Oct 30 18:03:36.634755 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:03:36.634721 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:03:54.535326 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:03:54.535253 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:04:01.893643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:01.893609 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 18:04:01.894049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:01.893801 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" containerID="cri-o://3e91242e16f43276e73773ed61eb014a2b02799edbbddbcb6d2b412e1b978510" gracePeriod=30 Oct 30 18:04:07.018660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.018629 4622 generic.go:356] "Generic (PLEG): container finished" podID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerID="3e91242e16f43276e73773ed61eb014a2b02799edbbddbcb6d2b412e1b978510" exitCode=0 Oct 30 18:04:07.018992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.018705 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" event={"ID":"1e2323cb-9feb-4803-92f1-8f2146ebdc43","Type":"ContainerDied","Data":"3e91242e16f43276e73773ed61eb014a2b02799edbbddbcb6d2b412e1b978510"} Oct 30 18:04:07.309592 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.309548 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 18:04:07.346464 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346433 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346674 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346475 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346674 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346541 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346674 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346633 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346674 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346665 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346694 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346759 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346793 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.346936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.346842 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\") pod \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\" (UID: \"1e2323cb-9feb-4803-92f1-8f2146ebdc43\") " Oct 30 18:04:07.347093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347037 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:07.347175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347156 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:07.347348 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347301 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:07.347396 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347359 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:07.347482 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347460 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:07.347645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347623 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.347645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347645 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.347831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347656 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.347831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347664 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.347831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.347672 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.349586 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.349561 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:07.349745 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.349662 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:07.349745 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.349690 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:07.350023 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.350005 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv" pod "1e2323cb-9feb-4803-92f1-8f2146ebdc43" (UID: "1e2323cb-9feb-4803-92f1-8f2146ebdc43"). InnerVolumeSpecName "kube-api-access-scnmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:07.448766 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.448735 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.448766 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.448765 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.448960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.448777 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2323cb-9feb-4803-92f1-8f2146ebdc43-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:07.448960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:07.448790 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scnmv\" (UniqueName: \"kubernetes.io/projected/1e2323cb-9feb-4803-92f1-8f2146ebdc43-kube-api-access-scnmv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:08.029014 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.028984 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" Oct 30 18:04:08.029697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.028983 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh" event={"ID":"1e2323cb-9feb-4803-92f1-8f2146ebdc43","Type":"ContainerDied","Data":"cfef9526a8fdd9721cdac92dd41c5f4fd0dfb646799b49c14c7f9b4b35b9c203"} Oct 30 18:04:08.029697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.029128 4622 scope.go:122] "RemoveContainer" containerID="3e91242e16f43276e73773ed61eb014a2b02799edbbddbcb6d2b412e1b978510" Oct 30 18:04:08.113617 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.113583 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 18:04:08.119454 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.119429 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-vmjlh"] Oct 30 18:04:08.376609 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:08.376571 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" path="/var/lib/kubelet/pods/1e2323cb-9feb-4803-92f1-8f2146ebdc43/volumes" Oct 30 18:04:14.873928 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:14.873888 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:04:24.534063 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:24.533989 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:04:25.444308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:25.444271 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 18:04:25.444627 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:25.444508 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" podUID="01a94584-506d-4b6d-81d8-88e8ddd408d4" containerName="maas-api" containerID="cri-o://2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07" gracePeriod=30 Oct 30 18:04:25.918145 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:25.918116 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 18:04:26.063997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.063884 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\") pod \"01a94584-506d-4b6d-81d8-88e8ddd408d4\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " Oct 30 18:04:26.064321 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.064293 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\" (UniqueName: \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\") pod \"01a94584-506d-4b6d-81d8-88e8ddd408d4\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " Oct 30 18:04:26.065474 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.065448 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\") pod \"01a94584-506d-4b6d-81d8-88e8ddd408d4\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " Oct 30 18:04:26.065673 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.065656 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\") pod \"01a94584-506d-4b6d-81d8-88e8ddd408d4\" (UID: \"01a94584-506d-4b6d-81d8-88e8ddd408d4\") " Oct 30 18:04:26.090295 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.084362 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca" pod "01a94584-506d-4b6d-81d8-88e8ddd408d4" (UID: "01a94584-506d-4b6d-81d8-88e8ddd408d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:26.118859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.118808 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs" pod "01a94584-506d-4b6d-81d8-88e8ddd408d4" (UID: "01a94584-506d-4b6d-81d8-88e8ddd408d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:26.119193 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.119172 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls" pod "01a94584-506d-4b6d-81d8-88e8ddd408d4" (UID: "01a94584-506d-4b6d-81d8-88e8ddd408d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:26.121874 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.121832 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp" pod "01a94584-506d-4b6d-81d8-88e8ddd408d4" (UID: "01a94584-506d-4b6d-81d8-88e8ddd408d4"). InnerVolumeSpecName "kube-api-access-szwqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:26.169005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.168968 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:26.169005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.169005 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwqp\" (UniqueName: \"kubernetes.io/projected/01a94584-506d-4b6d-81d8-88e8ddd408d4-kube-api-access-szwqp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:26.169275 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.169019 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01a94584-506d-4b6d-81d8-88e8ddd408d4-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:26.169275 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.169032 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01a94584-506d-4b6d-81d8-88e8ddd408d4-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:26.191074 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.191037 4622 generic.go:356] "Generic (PLEG): container finished" podID="01a94584-506d-4b6d-81d8-88e8ddd408d4" containerID="2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07" exitCode=0 Oct 30 18:04:26.191236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.191089 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" event={"ID":"01a94584-506d-4b6d-81d8-88e8ddd408d4","Type":"ContainerDied","Data":"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07"} Oct 30 18:04:26.191236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.191164 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" event={"ID":"01a94584-506d-4b6d-81d8-88e8ddd408d4","Type":"ContainerDied","Data":"8f5bc44025106b16fc373b74f5e6cce7eba23c682168551108fd7b7fc4670d25"} Oct 30 18:04:26.191236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.191100 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf" Oct 30 18:04:26.191236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.191196 4622 scope.go:122] "RemoveContainer" containerID="2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07" Oct 30 18:04:26.204297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.204275 4622 scope.go:122] "RemoveContainer" containerID="2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07" Oct 30 18:04:26.204756 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:26.204727 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07\": container with ID starting with 2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07 not found: ID does not exist" containerID="2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07" Oct 30 18:04:26.204860 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.204769 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07"} err="failed to get container status \"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07\": rpc error: code = NotFound desc = could not find container \"2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07\": container with ID starting with 2ba51ffa5b2e59e48ad8652e51eb440b30abaef0cb46b2240f68be81d7662d07 not found: ID does not exist" Oct 30 18:04:26.246197 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.246171 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 18:04:26.251993 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.251971 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-738845-6d86d89bc4-gdxzf"] Oct 30 18:04:26.377422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:26.377347 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a94584-506d-4b6d-81d8-88e8ddd408d4" path="/var/lib/kubelet/pods/01a94584-506d-4b6d-81d8-88e8ddd408d4/volumes" Oct 30 18:04:28.303689 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.303648 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 18:04:28.304158 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.303860 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" podUID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" containerName="maas-api" containerID="cri-o://1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad" gracePeriod=30 Oct 30 18:04:28.576120 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.576009 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:04:28.702210 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.702168 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 18:04:28.705355 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.705322 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\") pod \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " Oct 30 18:04:28.705700 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.705676 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\" (UniqueName: \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\") pod \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " Oct 30 18:04:28.705841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.705736 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\") pod \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " Oct 30 18:04:28.705841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.705773 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") pod \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\" (UID: \"a615c998-26e0-49ed-8878-6b1c0f7b1d9e\") " Oct 30 18:04:28.706661 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.706635 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca" pod "a615c998-26e0-49ed-8878-6b1c0f7b1d9e" (UID: "a615c998-26e0-49ed-8878-6b1c0f7b1d9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:28.707289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.707246 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:28.708949 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.708918 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs" pod "a615c998-26e0-49ed-8878-6b1c0f7b1d9e" (UID: "a615c998-26e0-49ed-8878-6b1c0f7b1d9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:28.709431 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.709403 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls" pod "a615c998-26e0-49ed-8878-6b1c0f7b1d9e" (UID: "a615c998-26e0-49ed-8878-6b1c0f7b1d9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:28.709556 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.709534 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46" pod "a615c998-26e0-49ed-8878-6b1c0f7b1d9e" (UID: "a615c998-26e0-49ed-8878-6b1c0f7b1d9e"). InnerVolumeSpecName "kube-api-access-w4f46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:28.808918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.808877 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4f46\" (UniqueName: \"kubernetes.io/projected/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-kube-api-access-w4f46\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:28.808918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.808919 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:28.809188 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:28.808934 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a615c998-26e0-49ed-8878-6b1c0f7b1d9e-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:29.220841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.220798 4622 generic.go:356] "Generic (PLEG): container finished" podID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" containerID="1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad" exitCode=0 Oct 30 18:04:29.221050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.220853 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" Oct 30 18:04:29.221050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.220886 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" event={"ID":"a615c998-26e0-49ed-8878-6b1c0f7b1d9e","Type":"ContainerDied","Data":"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad"} Oct 30 18:04:29.221050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.220947 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz" event={"ID":"a615c998-26e0-49ed-8878-6b1c0f7b1d9e","Type":"ContainerDied","Data":"ecf27539176b7309040ebb82a881a8c3e241def2ae5efa5669a8b06a00f1d8b3"} Oct 30 18:04:29.221050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.220986 4622 scope.go:122] "RemoveContainer" containerID="1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad" Oct 30 18:04:29.237134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.237110 4622 scope.go:122] "RemoveContainer" containerID="1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad" Oct 30 18:04:29.237460 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:29.237439 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad\": container with ID starting with 1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad not found: ID does not exist" containerID="1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad" Oct 30 18:04:29.237514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.237469 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad"} err="failed to get container status \"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad\": rpc error: code = NotFound desc = could not find container \"1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad\": container with ID starting with 1bc7ab5ea22462cc9d176e094ac39c64fee4525f820c13f14fef217c4c515bad not found: ID does not exist" Oct 30 18:04:29.280789 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.279288 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 18:04:29.292688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.292653 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-455e4a-65c889556c-tkpxz"] Oct 30 18:04:29.911098 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:29.911058 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerName="main" containerID="cri-o://21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9" gracePeriod=45 Oct 30 18:04:30.185387 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.185364 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:04:30.230189 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.230160 4622 generic.go:356] "Generic (PLEG): container finished" podID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerID="21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9" exitCode=0 Oct 30 18:04:30.230391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.230201 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" Oct 30 18:04:30.230391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.230240 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" event={"ID":"1aec6088-65d8-406f-a00f-3c51adfc400a","Type":"ContainerDied","Data":"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9"} Oct 30 18:04:30.230391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.230299 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" event={"ID":"1aec6088-65d8-406f-a00f-3c51adfc400a","Type":"ContainerDied","Data":"5a3697a4de31f8bc6c051f3494db909429d3626d34d9ca2a47e5251961b51bc1"} Oct 30 18:04:30.230391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.230329 4622 scope.go:122] "RemoveContainer" containerID="21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9" Oct 30 18:04:30.242470 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.242447 4622 scope.go:122] "RemoveContainer" containerID="21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9" Oct 30 18:04:30.242792 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:30.242762 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9\": container with ID starting with 21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9 not found: ID does not exist" containerID="21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9" Oct 30 18:04:30.242898 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.242803 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9"} err="failed to get container status \"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9\": rpc error: code = NotFound desc = could not find container \"21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9\": container with ID starting with 21a2fc2bc718c0e8ed05f34cc7083f94257f2b262825fc74dd74a2ac5466bbf9 not found: ID does not exist" Oct 30 18:04:30.278239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278212 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278271 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278298 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278346 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278448 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\" (UniqueName: \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278502 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\" (UniqueName: \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\") pod \"1aec6088-65d8-406f-a00f-3c51adfc400a\" (UID: \"1aec6088-65d8-406f-a00f-3c51adfc400a\") " Oct 30 18:04:30.278549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278445 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:30.278702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278627 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:30.278702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.278662 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:30.279090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.279068 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.279090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.279090 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.279323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.279098 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.280691 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.280668 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:30.280691 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.280680 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:30.280859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.280743 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6" pod "1aec6088-65d8-406f-a00f-3c51adfc400a" (UID: "1aec6088-65d8-406f-a00f-3c51adfc400a"). InnerVolumeSpecName "kube-api-access-pcrn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:30.375707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.375676 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" path="/var/lib/kubelet/pods/a615c998-26e0-49ed-8878-6b1c0f7b1d9e/volumes" Oct 30 18:04:30.380216 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.380192 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aec6088-65d8-406f-a00f-3c51adfc400a-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.380312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.380223 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrn6\" (UniqueName: \"kubernetes.io/projected/1aec6088-65d8-406f-a00f-3c51adfc400a-kube-api-access-pcrn6\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.380312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.380241 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aec6088-65d8-406f-a00f-3c51adfc400a-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:30.575537 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.575503 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:04:30.582183 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:30.582070 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz"] Oct 30 18:04:31.113284 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:31.113032 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-23c1e8/rate-test-model-23c1e8-kserve-7cdc6fdc57-28dkz" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:04:32.375963 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:32.375926 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" path="/var/lib/kubelet/pods/1aec6088-65d8-406f-a00f-3c51adfc400a/volumes" Oct 30 18:04:43.656779 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.656734 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" podUID="59a85f43-178e-4a63-9712-30ff98dd8d43" containerName="main" containerID="cri-o://63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3" gracePeriod=45 Oct 30 18:04:43.938251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.938223 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:04:43.987877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.987848 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.987897 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.987934 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.987962 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988018 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988059 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\" (UniqueName: \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\") pod \"59a85f43-178e-4a63-9712-30ff98dd8d43\" (UID: \"59a85f43-178e-4a63-9712-30ff98dd8d43\") " Oct 30 18:04:43.988449 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988232 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:43.988449 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988418 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:43.988774 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988752 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:43.988774 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988775 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:43.988981 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.988871 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:43.992202 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.990968 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:04:43.992202 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.991037 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "kube-api-access-hk8qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:43.992650 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:43.992630 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm" pod "59a85f43-178e-4a63-9712-30ff98dd8d43" (UID: "59a85f43-178e-4a63-9712-30ff98dd8d43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:44.089643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.089611 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:44.089643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.089639 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a85f43-178e-4a63-9712-30ff98dd8d43-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:44.089810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.089648 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a85f43-178e-4a63-9712-30ff98dd8d43-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:44.089810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.089658 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8qt\" (UniqueName: \"kubernetes.io/projected/59a85f43-178e-4a63-9712-30ff98dd8d43-kube-api-access-hk8qt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:44.347005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.346969 4622 generic.go:356] "Generic (PLEG): container finished" podID="59a85f43-178e-4a63-9712-30ff98dd8d43" containerID="63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3" exitCode=0 Oct 30 18:04:44.347214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.347030 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" Oct 30 18:04:44.347214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.347105 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" event={"ID":"59a85f43-178e-4a63-9712-30ff98dd8d43","Type":"ContainerDied","Data":"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3"} Oct 30 18:04:44.347214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.347150 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt" event={"ID":"59a85f43-178e-4a63-9712-30ff98dd8d43","Type":"ContainerDied","Data":"81940b76a811afabe7f50108042c09cbf261545fa1ce28b8ef2b5083822e33a2"} Oct 30 18:04:44.347214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.347171 4622 scope.go:122] "RemoveContainer" containerID="63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3" Oct 30 18:04:44.359998 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.359979 4622 scope.go:122] "RemoveContainer" containerID="63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3" Oct 30 18:04:44.360313 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:44.360252 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3\": container with ID starting with 63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3 not found: ID does not exist" containerID="63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3" Oct 30 18:04:44.360377 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.360326 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3"} err="failed to get container status \"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3\": rpc error: code = NotFound desc = could not find container \"63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3\": container with ID starting with 63a9ed3d68cd51fcc31837d9516ff888f20a4f1be2ab9df89e3b4bcca1b25de3 not found: ID does not exist" Oct 30 18:04:44.454367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.454321 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:04:44.474025 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:44.473996 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-d207f8/rate-test-model-d207f8-kserve-7c89f587df-tlhnt"] Oct 30 18:04:46.379849 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:46.379814 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a85f43-178e-4a63-9712-30ff98dd8d43" path="/var/lib/kubelet/pods/59a85f43-178e-4a63-9712-30ff98dd8d43/volumes" Oct 30 18:04:51.713798 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:51.713765 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 18:04:51.714213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:51.713955 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" containerID="cri-o://17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a" gracePeriod=30 Oct 30 18:04:53.355982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:53.355933 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 18:04:53.357134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:53.356709 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" containerID="cri-o://c6741dd5ef212932efd3a569e3779233b90db5c1753b6ffc8c50259dd126b993" gracePeriod=30 Oct 30 18:04:54.534250 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:54.534200 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:04:56.487774 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.487744 4622 generic.go:356] "Generic (PLEG): container finished" podID="685cc8ad-996f-4813-a663-0b9e875fe978" containerID="2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2" exitCode=0 Oct 30 18:04:56.488310 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.487816 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerDied","Data":"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2"} Oct 30 18:04:56.488310 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.488205 4622 scope.go:122] "RemoveContainer" containerID="2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2" Oct 30 18:04:56.489825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.489793 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:56.489965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.489874 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:56.489965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:56.489924 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:57.007306 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.007229 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 18:04:57.104880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.104845 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.104939 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.104976 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105003 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105284 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105074 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105284 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105109 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105415 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105315 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:57.105415 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105392 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:57.105415 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105403 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:57.105576 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105171 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105576 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105471 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:57.105576 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105480 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105576 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105540 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\") pod \"2b08100e-eb64-4016-983b-64acf8052259\" (UID: \"2b08100e-eb64-4016-983b-64acf8052259\") " Oct 30 18:04:57.105802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.105670 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:57.106195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.106175 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.106195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.106195 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b08100e-eb64-4016-983b-64acf8052259-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.106402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.106204 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.106402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.106213 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.106402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.106221 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.107250 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.107220 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:57.107515 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.107496 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:57.107942 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.107922 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:57.108008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.107978 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp" pod "2b08100e-eb64-4016-983b-64acf8052259" (UID: "2b08100e-eb64-4016-983b-64acf8052259"). InnerVolumeSpecName "kube-api-access-vjmrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:57.207743 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.207703 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.207743 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.207739 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmrp\" (UniqueName: \"kubernetes.io/projected/2b08100e-eb64-4016-983b-64acf8052259-kube-api-access-vjmrp\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.207960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.207754 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b08100e-eb64-4016-983b-64acf8052259-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.207960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.207767 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b08100e-eb64-4016-983b-64acf8052259-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:57.497452 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.497414 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerStarted","Data":"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493"} Oct 30 18:04:57.499225 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.499202 4622 generic.go:356] "Generic (PLEG): container finished" podID="2b08100e-eb64-4016-983b-64acf8052259" containerID="17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a" exitCode=0 Oct 30 18:04:57.499348 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.499245 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" Oct 30 18:04:57.499348 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.499294 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerDied","Data":"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a"} Oct 30 18:04:57.499469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.499356 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr" event={"ID":"2b08100e-eb64-4016-983b-64acf8052259","Type":"ContainerDied","Data":"aab56034fd943a5cddf0e932630dfaeb3a007f4ad0c9db46b11adab3ae5c9e85"} Oct 30 18:04:57.499469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.499388 4622 scope.go:122] "RemoveContainer" containerID="17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a" Oct 30 18:04:57.510345 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.510330 4622 scope.go:122] "RemoveContainer" containerID="8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340" Oct 30 18:04:57.519664 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.519642 4622 scope.go:122] "RemoveContainer" containerID="17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a" Oct 30 18:04:57.519928 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:57.519909 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a\": container with ID starting with 17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a not found: ID does not exist" containerID="17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a" Oct 30 18:04:57.519984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.519937 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a"} err="failed to get container status \"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a\": rpc error: code = NotFound desc = could not find container \"17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a\": container with ID starting with 17c6760da7dbefae7dc2c69225b398e869ce135a7bb95158f10a4d4056bffd9a not found: ID does not exist" Oct 30 18:04:57.519984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.519955 4622 scope.go:122] "RemoveContainer" containerID="8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340" Oct 30 18:04:57.520204 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:04:57.520184 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340\": container with ID starting with 8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340 not found: ID does not exist" containerID="8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340" Oct 30 18:04:57.520297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.520210 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340"} err="failed to get container status \"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340\": rpc error: code = NotFound desc = could not find container \"8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340\": container with ID starting with 8a0f2f49ba70c8860b02ee55966a8a1271367bb853c1fc5bda6b4378dfb49340 not found: ID does not exist" Oct 30 18:04:57.596166 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.596132 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 18:04:57.599845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:57.599823 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-kwznr"] Oct 30 18:04:58.352602 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.352572 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:04:58.365419 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.365389 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 18:04:58.365562 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.365439 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 18:04:58.376045 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.376012 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b08100e-eb64-4016-983b-64acf8052259" path="/var/lib/kubelet/pods/2b08100e-eb64-4016-983b-64acf8052259/volumes" Oct 30 18:04:58.509839 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.509789 4622 generic.go:356] "Generic (PLEG): container finished" podID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerID="c6741dd5ef212932efd3a569e3779233b90db5c1753b6ffc8c50259dd126b993" exitCode=0 Oct 30 18:04:58.510239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.509867 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerDied","Data":"c6741dd5ef212932efd3a569e3779233b90db5c1753b6ffc8c50259dd126b993"} Oct 30 18:04:58.510239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.509926 4622 scope.go:122] "RemoveContainer" containerID="b9f5a9a05d9d4987ea8c2e4488ec968c14afa0c27eebe4b3db9cc6568e44ed88" Oct 30 18:04:58.682987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.682961 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 18:04:58.828251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828216 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828457 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828334 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828457 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828388 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828566 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828459 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828566 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828512 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828680 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828578 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828680 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828641 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828680 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828650 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.828846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828737 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828745 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.828846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828782 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\") pod \"efde37f6-504c-4dd5-9764-65910dc63ca2\" (UID: \"efde37f6-504c-4dd5-9764-65910dc63ca2\") " Oct 30 18:04:58.828846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828797 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:04:58.829082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828906 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.829082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.828944 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.829304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.829276 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.829304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.829298 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efde37f6-504c-4dd5-9764-65910dc63ca2-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.829461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.829312 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.829461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.829325 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.829461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.829333 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.830877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.830850 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:58.830877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.830868 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:04:58.831175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.831154 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:58.831230 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.831189 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl" pod "efde37f6-504c-4dd5-9764-65910dc63ca2" (UID: "efde37f6-504c-4dd5-9764-65910dc63ca2"). InnerVolumeSpecName "kube-api-access-x8crl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:58.923644 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.923577 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 18:04:58.930939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.930916 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8crl\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-kube-api-access-x8crl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.930939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.930939 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.931082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.930950 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.931082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.930958 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efde37f6-504c-4dd5-9764-65910dc63ca2-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:04:58.940044 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:58.940017 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 18:04:59.378199 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.378156 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 18:04:59.378444 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.378228 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 18:04:59.378741 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.378718 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 18:04:59.379013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.378988 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" podUID="116cf21b-2633-4a7c-aebf-3d7855282cce" containerName="maas-api" containerID="cri-o://da4d9299b0210a6f8f18546d3f1b2e30b5e3989a640f8a514a51d152bf42e161" gracePeriod=30 Oct 30 18:04:59.496929 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.496895 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:04:59.584328 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.583925 4622 generic.go:356] "Generic (PLEG): container finished" podID="116cf21b-2633-4a7c-aebf-3d7855282cce" containerID="da4d9299b0210a6f8f18546d3f1b2e30b5e3989a640f8a514a51d152bf42e161" exitCode=0 Oct 30 18:04:59.584328 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.584032 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" event={"ID":"116cf21b-2633-4a7c-aebf-3d7855282cce","Type":"ContainerDied","Data":"da4d9299b0210a6f8f18546d3f1b2e30b5e3989a640f8a514a51d152bf42e161"} Oct 30 18:04:59.589458 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.588341 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" event={"ID":"efde37f6-504c-4dd5-9764-65910dc63ca2","Type":"ContainerDied","Data":"d00a2e804369d9a2d8e38cd9dcb3a7627a15c72828a3bb363352b6092cd63735"} Oct 30 18:04:59.589458 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.588404 4622 scope.go:122] "RemoveContainer" containerID="c6741dd5ef212932efd3a569e3779233b90db5c1753b6ffc8c50259dd126b993" Oct 30 18:04:59.589458 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.588433 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj" Oct 30 18:04:59.684430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.684339 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 18:04:59.688916 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.688869 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-738845-openshift-default-85b4949f49-7s6hj"] Oct 30 18:04:59.881574 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.881541 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 18:04:59.996673 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.996574 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\") pod \"116cf21b-2633-4a7c-aebf-3d7855282cce\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " Oct 30 18:04:59.996859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.996734 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\" (UniqueName: \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\") pod \"116cf21b-2633-4a7c-aebf-3d7855282cce\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " Oct 30 18:04:59.996859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.996761 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\") pod \"116cf21b-2633-4a7c-aebf-3d7855282cce\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " Oct 30 18:04:59.996859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.996809 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\") pod \"116cf21b-2633-4a7c-aebf-3d7855282cce\" (UID: \"116cf21b-2633-4a7c-aebf-3d7855282cce\") " Oct 30 18:04:59.997940 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:04:59.997909 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca" pod "116cf21b-2633-4a7c-aebf-3d7855282cce" (UID: "116cf21b-2633-4a7c-aebf-3d7855282cce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:00.001450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.001426 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls" pod "116cf21b-2633-4a7c-aebf-3d7855282cce" (UID: "116cf21b-2633-4a7c-aebf-3d7855282cce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:00.001578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.001444 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz" pod "116cf21b-2633-4a7c-aebf-3d7855282cce" (UID: "116cf21b-2633-4a7c-aebf-3d7855282cce"). InnerVolumeSpecName "kube-api-access-qxqlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:00.002942 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.002920 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs" pod "116cf21b-2633-4a7c-aebf-3d7855282cce" (UID: "116cf21b-2633-4a7c-aebf-3d7855282cce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:00.098751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.098631 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116cf21b-2633-4a7c-aebf-3d7855282cce-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.098751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.098671 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqlz\" (UniqueName: \"kubernetes.io/projected/116cf21b-2633-4a7c-aebf-3d7855282cce-kube-api-access-qxqlz\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.098751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.098687 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.098751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.098701 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116cf21b-2633-4a7c-aebf-3d7855282cce-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:00.355293 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.353665 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 18:05:00.355293 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.353732 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 18:05:00.375998 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.375969 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" path="/var/lib/kubelet/pods/efde37f6-504c-4dd5-9764-65910dc63ca2/volumes" Oct 30 18:05:00.596438 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.596405 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" Oct 30 18:05:00.596887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.596404 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6" event={"ID":"116cf21b-2633-4a7c-aebf-3d7855282cce","Type":"ContainerDied","Data":"98f0a8447cfb817017157546fc980678605dd6f1e1b563707a23acb189ffba6b"} Oct 30 18:05:00.596887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.596540 4622 scope.go:122] "RemoveContainer" containerID="da4d9299b0210a6f8f18546d3f1b2e30b5e3989a640f8a514a51d152bf42e161" Oct 30 18:05:00.668384 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.668351 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 18:05:00.679021 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:00.678993 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fab25f-85c475979-fjdd6"] Oct 30 18:05:01.365401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:01.365362 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 18:05:01.365604 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:01.365451 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 18:05:02.013524 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:02.002932 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:05:02.013524 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:02.003173 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" containerID="cri-o://59d297ad68bd4537622ef7cbb67a7bd40f21efa961e2f1e639b80675fa594c8f" gracePeriod=30 Oct 30 18:05:02.354740 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:02.354700 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 18:05:02.355064 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:02.355018 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 18:05:02.376049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:02.376023 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116cf21b-2633-4a7c-aebf-3d7855282cce" path="/var/lib/kubelet/pods/116cf21b-2633-4a7c-aebf-3d7855282cce/volumes" Oct 30 18:05:03.359430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:03.359391 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:05:03.360935 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:03.360909 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:05:04.366025 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:04.365995 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 18:05:07.391207 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.391164 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 18:05:07.391721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.391362 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" containerID="cri-o://4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46" gracePeriod=30 Oct 30 18:05:07.661074 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.661043 4622 generic.go:356] "Generic (PLEG): container finished" podID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerID="59d297ad68bd4537622ef7cbb67a7bd40f21efa961e2f1e639b80675fa594c8f" exitCode=0 Oct 30 18:05:07.661235 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.661187 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerDied","Data":"59d297ad68bd4537622ef7cbb67a7bd40f21efa961e2f1e639b80675fa594c8f"} Oct 30 18:05:07.661235 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.661230 4622 scope.go:122] "RemoveContainer" containerID="eabd4a05acbbf90f781eea3a861e4533a05a501b675b4d1af02393ae6365b619" Oct 30 18:05:07.765294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.762481 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:05:07.782349 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782316 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782497 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782475 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782512 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782592 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782575 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:07.782642 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782582 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782642 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782612 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782752 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782656 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782811 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782688 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782811 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782768 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.782811 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782799 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.782974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782837 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.782974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.782909 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\") pod \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\" (UID: \"d3a5f1eb-9331-425d-9e91-df7a100d4a3d\") " Oct 30 18:05:07.783101 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783025 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.783168 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783102 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.783434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783415 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.783434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783434 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.783595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783443 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.783595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783456 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.783595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.783468 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.784822 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.784795 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:07.784987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.784966 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:07.785330 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.785310 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "kube-api-access-kdd69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:07.785382 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.785346 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token" pod "d3a5f1eb-9331-425d-9e91-df7a100d4a3d" (UID: "d3a5f1eb-9331-425d-9e91-df7a100d4a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:07.885004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.884969 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.885004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.885006 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.885196 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.885021 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:07.885196 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:07.885036 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdd69\" (UniqueName: \"kubernetes.io/projected/d3a5f1eb-9331-425d-9e91-df7a100d4a3d-kube-api-access-kdd69\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:08.668854 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:08.668775 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" Oct 30 18:05:08.668854 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:08.668821 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46" event={"ID":"d3a5f1eb-9331-425d-9e91-df7a100d4a3d","Type":"ContainerDied","Data":"03deed6b13ae84f7f967709de83cb5863b2ebeec522c63428a28e29152ec3abd"} Oct 30 18:05:08.669369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:08.668879 4622 scope.go:122] "RemoveContainer" containerID="59d297ad68bd4537622ef7cbb67a7bd40f21efa961e2f1e639b80675fa594c8f" Oct 30 18:05:08.730431 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:08.730405 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:05:08.741917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:08.741888 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b5cbb4-openshift-default-6dc5fd5b9c-zqh46"] Oct 30 18:05:09.366980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:09.366924 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" containerID="cri-o://47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805" gracePeriod=25 Oct 30 18:05:10.374856 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:10.374819 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" path="/var/lib/kubelet/pods/d3a5f1eb-9331-425d-9e91-df7a100d4a3d/volumes" Oct 30 18:05:12.687202 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.687175 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 18:05:12.701825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.701796 4622 generic.go:356] "Generic (PLEG): container finished" podID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerID="4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46" exitCode=0 Oct 30 18:05:12.701987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.701847 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" Oct 30 18:05:12.701987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.701866 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" event={"ID":"7bd1506e-d8a3-432d-8180-60377fe3c5a1","Type":"ContainerDied","Data":"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46"} Oct 30 18:05:12.701987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.701919 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt" event={"ID":"7bd1506e-d8a3-432d-8180-60377fe3c5a1","Type":"ContainerDied","Data":"7d394cf3d64dbf0c340131f2b5ffdeb7c53ab3fb347c122ed049957fd524f996"} Oct 30 18:05:12.701987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.701951 4622 scope.go:122] "RemoveContainer" containerID="4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46" Oct 30 18:05:12.712188 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.712166 4622 scope.go:122] "RemoveContainer" containerID="4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46" Oct 30 18:05:12.712493 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:12.712466 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46\": container with ID starting with 4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46 not found: ID does not exist" containerID="4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46" Oct 30 18:05:12.712618 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.712503 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46"} err="failed to get container status \"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46\": rpc error: code = NotFound desc = could not find container \"4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46\": container with ID starting with 4f0010efdda16dc99e42c3014526c5f720e19aa0ac38035be70f82d8bb003c46 not found: ID does not exist" Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.749953 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750006 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750038 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750064 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750152 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750179 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750272 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750321 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.750473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750370 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\") pod \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\" (UID: \"7bd1506e-d8a3-432d-8180-60377fe3c5a1\") " Oct 30 18:05:12.751070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750484 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.751070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750921 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.751070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.750951 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.751248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.751138 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.751248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.751157 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.751248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.751172 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.751248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.751199 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:12.751762 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.751577 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.752771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.752737 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "kube-api-access-7jst4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:12.753757 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.753734 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:12.754013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.753991 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:12.755207 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.755186 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token" pod "7bd1506e-d8a3-432d-8180-60377fe3c5a1" (UID: "7bd1506e-d8a3-432d-8180-60377fe3c5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:12.852084 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852053 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.852084 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852084 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.852304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852094 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.852304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852111 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.852304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852120 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jst4\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-kube-api-access-7jst4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:12.852304 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:12.852128 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd1506e-d8a3-432d-8180-60377fe3c5a1-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:13.055527 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:13.055492 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 18:05:13.063297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:13.063246 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fab25f-openshift-default-787bdf685f-fdtvt"] Oct 30 18:05:13.974509 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:13.974461 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" containerID="cri-o://5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de" gracePeriod=45 Oct 30 18:05:13.987165 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:13.987074 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" containerID="cri-o://a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559" gracePeriod=45 Oct 30 18:05:14.094718 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.094683 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 30 18:05:14.316527 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.316499 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 18:05:14.319826 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.319805 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 18:05:14.337317 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.337250 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:14.384092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.384055 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.387034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.387004 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:14.387803 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.387770 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" path="/var/lib/kubelet/pods/7bd1506e-d8a3-432d-8180-60377fe3c5a1/volumes" Oct 30 18:05:14.388697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.388672 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.388877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.388845 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.388982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.388966 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.389092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.389073 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\" (UniqueName: \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.389171 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.389126 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.389231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.389172 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\" (UniqueName: \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.389765 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.389737 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.391370 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.391343 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.392294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.392273 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.392386 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.392347 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.392693 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.392667 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.392887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.392398 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.393768 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.393745 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.394461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.394016 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\") pod \"5c442a99-d65a-49df-a1cf-79938df4f4ac\" (UID: \"5c442a99-d65a-49df-a1cf-79938df4f4ac\") " Oct 30 18:05:14.394549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.394531 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\") pod \"f8c80672-f796-4050-bcfe-c274eb4c805d\" (UID: \"f8c80672-f796-4050-bcfe-c274eb4c805d\") " Oct 30 18:05:14.394689 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.394055 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.394689 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.394181 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.394930 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.394814 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.395340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395317 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.395653 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395366 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.395775 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395763 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.395877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395866 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c80672-f796-4050-bcfe-c274eb4c805d-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.395964 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395947 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.395964 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395966 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.396360 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395978 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.396360 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.395991 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.396360 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.396004 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c442a99-d65a-49df-a1cf-79938df4f4ac-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.399297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.399249 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw" pod "f8c80672-f796-4050-bcfe-c274eb4c805d" (UID: "f8c80672-f796-4050-bcfe-c274eb4c805d"). InnerVolumeSpecName "kube-api-access-mm9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.403285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.399810 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "kube-api-access-cz862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.403285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.402021 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs" pod "5c442a99-d65a-49df-a1cf-79938df4f4ac" (UID: "5c442a99-d65a-49df-a1cf-79938df4f4ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:14.496865 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.496819 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz862\" (UniqueName: \"kubernetes.io/projected/5c442a99-d65a-49df-a1cf-79938df4f4ac-kube-api-access-cz862\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.496865 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.496853 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c442a99-d65a-49df-a1cf-79938df4f4ac-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.496865 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.496867 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9mw\" (UniqueName: \"kubernetes.io/projected/f8c80672-f796-4050-bcfe-c274eb4c805d-kube-api-access-mm9mw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.497165 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.496882 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c80672-f796-4050-bcfe-c274eb4c805d-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.734151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.734046 4622 generic.go:356] "Generic (PLEG): container finished" podID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerID="a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559" exitCode=0 Oct 30 18:05:14.734359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.734167 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" event={"ID":"f8c80672-f796-4050-bcfe-c274eb4c805d","Type":"ContainerDied","Data":"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559"} Oct 30 18:05:14.734359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.734202 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" event={"ID":"f8c80672-f796-4050-bcfe-c274eb4c805d","Type":"ContainerDied","Data":"08bdf243c68245e6d60869022597a709af7047b614f818178f801eb62565b67d"} Oct 30 18:05:14.734359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.734225 4622 scope.go:122] "RemoveContainer" containerID="a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559" Oct 30 18:05:14.734359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.734359 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb" Oct 30 18:05:14.754450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.741769 4622 generic.go:356] "Generic (PLEG): container finished" podID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerID="5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de" exitCode=0 Oct 30 18:05:14.754450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.741830 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" event={"ID":"5c442a99-d65a-49df-a1cf-79938df4f4ac","Type":"ContainerDied","Data":"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de"} Oct 30 18:05:14.754450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.741861 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" event={"ID":"5c442a99-d65a-49df-a1cf-79938df4f4ac","Type":"ContainerDied","Data":"20bf41e1ad935bf53013e744c497b100cbff0de32827c4dc84f376c352e2ece8"} Oct 30 18:05:14.754450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.741920 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4" Oct 30 18:05:14.769500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.769479 4622 scope.go:122] "RemoveContainer" containerID="a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559" Oct 30 18:05:14.770031 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:14.770003 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559\": container with ID starting with a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559 not found: ID does not exist" containerID="a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559" Oct 30 18:05:14.770153 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.770044 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559"} err="failed to get container status \"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559\": rpc error: code = NotFound desc = could not find container \"a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559\": container with ID starting with a3cc65a61406549c62e19895319ce2a7117e2df66d7187c14ca6f4740a038559 not found: ID does not exist" Oct 30 18:05:14.770153 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.770069 4622 scope.go:122] "RemoveContainer" containerID="5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de" Oct 30 18:05:14.784308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.784273 4622 scope.go:122] "RemoveContainer" containerID="5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de" Oct 30 18:05:14.784730 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:14.784707 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de\": container with ID starting with 5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de not found: ID does not exist" containerID="5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de" Oct 30 18:05:14.784835 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.784740 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de"} err="failed to get container status \"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de\": rpc error: code = NotFound desc = could not find container \"5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de\": container with ID starting with 5d9e9d9ee8fc9901af4025ab18c552b1cd0c693627ae56a896b642785ef684de not found: ID does not exist" Oct 30 18:05:14.806930 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.806902 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 18:05:14.817800 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.817774 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/premium-w3-306aed-kserve-855989f9c8-652rb"] Oct 30 18:05:14.849624 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.849595 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 18:05:14.849739 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:14.849638 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-306aed/distinct-2-w3-306aed-kserve-85bc548ff4-wqgt4"] Oct 30 18:05:16.374971 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:16.374935 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" path="/var/lib/kubelet/pods/5c442a99-d65a-49df-a1cf-79938df4f4ac/volumes" Oct 30 18:05:16.375410 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:16.375395 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" path="/var/lib/kubelet/pods/f8c80672-f796-4050-bcfe-c274eb4c805d/volumes" Oct 30 18:05:22.659159 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:22.659123 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 18:05:22.659623 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:22.659423 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" containerID="cri-o://7210d8d3498d1f4059584bc10cdef6f87cc96ac0ed16cabd1842935bf3d71bb1" gracePeriod=30 Oct 30 18:05:24.337171 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:24.337131 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:05:24.533299 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:24.533241 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:05:27.880414 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.880379 4622 generic.go:356] "Generic (PLEG): container finished" podID="94d32316-94a5-4484-b67a-387c737461e3" containerID="7210d8d3498d1f4059584bc10cdef6f87cc96ac0ed16cabd1842935bf3d71bb1" exitCode=0 Oct 30 18:05:27.880844 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.880458 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerDied","Data":"7210d8d3498d1f4059584bc10cdef6f87cc96ac0ed16cabd1842935bf3d71bb1"} Oct 30 18:05:27.880844 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.880538 4622 scope.go:122] "RemoveContainer" containerID="6d0f3cfe18082d56e5acb5f1885a0dbfcdc4bae1a1b50c20a0140f959cff011d" Oct 30 18:05:27.946784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.946762 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 18:05:27.987014 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.986988 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987146 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987027 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987146 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987063 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987146 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987130 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987181 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987203 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987252 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987296 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987319 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:27.987373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987365 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987421 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\") pod \"94d32316-94a5-4484-b67a-387c737461e3\" (UID: \"94d32316-94a5-4484-b67a-387c737461e3\") " Oct 30 18:05:27.987712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987520 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:27.987712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987629 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:27.987712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987656 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:27.987894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987860 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:27.987894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987876 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:27.987894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987884 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d32316-94a5-4484-b67a-387c737461e3-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:27.987894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987892 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:27.988048 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.987900 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:27.989787 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.989763 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:27.989881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.989764 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:27.990097 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.990079 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "kube-api-access-qmn7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:27.990150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:27.990110 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token" pod "94d32316-94a5-4484-b67a-387c737461e3" (UID: "94d32316-94a5-4484-b67a-387c737461e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:28.089231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.089197 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d32316-94a5-4484-b67a-387c737461e3-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.089231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.089227 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.089231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.089236 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmn7q\" (UniqueName: \"kubernetes.io/projected/94d32316-94a5-4484-b67a-387c737461e3-kube-api-access-qmn7q\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.089480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.089246 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d32316-94a5-4484-b67a-387c737461e3-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:28.898291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.897286 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" event={"ID":"94d32316-94a5-4484-b67a-387c737461e3","Type":"ContainerDied","Data":"c75029408ce9d15847768a8491ae3c0d0ec5d4df1ac5bd51a91605f70280279a"} Oct 30 18:05:28.898291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.897357 4622 scope.go:122] "RemoveContainer" containerID="7210d8d3498d1f4059584bc10cdef6f87cc96ac0ed16cabd1842935bf3d71bb1" Oct 30 18:05:28.898291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.897515 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc" Oct 30 18:05:28.952397 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.952368 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 18:05:28.964243 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:28.964215 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-23c1e8-openshift-default-786474b749-qgtbc"] Oct 30 18:05:29.658442 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.658410 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 18:05:29.719346 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.719314 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\") pod \"3b362a8b-9719-4a90-ba33-81417c635064\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " Oct 30 18:05:29.719530 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.719433 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\") pod \"3b362a8b-9719-4a90-ba33-81417c635064\" (UID: \"3b362a8b-9719-4a90-ba33-81417c635064\") " Oct 30 18:05:29.719719 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.719696 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume" pod "3b362a8b-9719-4a90-ba33-81417c635064" (UID: "3b362a8b-9719-4a90-ba33-81417c635064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:29.719894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.719870 4622 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b362a8b-9719-4a90-ba33-81417c635064-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:29.723546 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.723516 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44" pod "3b362a8b-9719-4a90-ba33-81417c635064" (UID: "3b362a8b-9719-4a90-ba33-81417c635064"). InnerVolumeSpecName "kube-api-access-j9q44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:29.821495 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.821465 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/3b362a8b-9719-4a90-ba33-81417c635064-kube-api-access-j9q44\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:29.906491 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.906422 4622 generic.go:356] "Generic (PLEG): container finished" podID="3b362a8b-9719-4a90-ba33-81417c635064" containerID="47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805" exitCode=0 Oct 30 18:05:29.906491 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.906474 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" Oct 30 18:05:29.906934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.906503 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" event={"ID":"3b362a8b-9719-4a90-ba33-81417c635064","Type":"ContainerDied","Data":"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805"} Oct 30 18:05:29.906934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.906572 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt" event={"ID":"3b362a8b-9719-4a90-ba33-81417c635064","Type":"ContainerDied","Data":"a1c412dbf7a7db18fbd4de1b156c2dbf82bbd3c8b355407ca0c0bc807672060e"} Oct 30 18:05:29.906934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.906594 4622 scope.go:122] "RemoveContainer" containerID="47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805" Oct 30 18:05:29.917273 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.917237 4622 scope.go:122] "RemoveContainer" containerID="47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805" Oct 30 18:05:29.917555 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:29.917533 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805\": container with ID starting with 47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805 not found: ID does not exist" containerID="47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805" Oct 30 18:05:29.917618 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.917564 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805"} err="failed to get container status \"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805\": rpc error: code = NotFound desc = could not find container \"47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805\": container with ID starting with 47d439db7a2dd3b8b52248ad092807ad81f6091521b9a98439b4ae89794d6805 not found: ID does not exist" Oct 30 18:05:29.974349 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.974327 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 18:05:29.986774 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:29.986752 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-306aed-79c686b8cf-n8xxt"] Oct 30 18:05:30.375311 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:30.375279 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b362a8b-9719-4a90-ba33-81417c635064" path="/var/lib/kubelet/pods/3b362a8b-9719-4a90-ba33-81417c635064/volumes" Oct 30 18:05:30.375753 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:30.375739 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d32316-94a5-4484-b67a-387c737461e3" path="/var/lib/kubelet/pods/94d32316-94a5-4484-b67a-387c737461e3/volumes" Oct 30 18:05:40.414013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:40.413974 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:05:40.414810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:40.414770 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" containerID="cri-o://b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2" gracePeriod=30 Oct 30 18:05:44.056195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:44.056155 4622 patch_prober.go:29] interesting pod/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:05:44.056732 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:44.056229 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:05:45.689664 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.689602 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:05:45.760583 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760547 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.760754 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760627 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.760754 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760684 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.760885 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760764 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.760885 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760869 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.761011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760912 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.761011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760956 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:45.761011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.760971 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.761011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761002 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.761232 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761025 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\") pod \"f1677362-ae19-4863-9ddb-0ea0a6841a16\" (UID: \"f1677362-ae19-4863-9ddb-0ea0a6841a16\") " Oct 30 18:05:45.761357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761330 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:45.761357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761343 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:45.761484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761460 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:45.761720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761693 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.761852 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761729 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.761852 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761744 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1677362-ae19-4863-9ddb-0ea0a6841a16-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.761852 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761757 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.761852 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.761701 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:45.763313 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.763280 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:45.763453 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.763316 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "kube-api-access-qfzlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:45.763825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.763792 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:05:45.763946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.763888 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token" pod "f1677362-ae19-4863-9ddb-0ea0a6841a16" (UID: "f1677362-ae19-4863-9ddb-0ea0a6841a16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:45.862600 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.862573 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzlk\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-kube-api-access-qfzlk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.862600 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.862599 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.862764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.862609 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.862764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.862619 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:45.862764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:45.862627 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1677362-ae19-4863-9ddb-0ea0a6841a16-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:46.057147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.057111 4622 generic.go:356] "Generic (PLEG): container finished" podID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerID="b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2" exitCode=0 Oct 30 18:05:46.057332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.057167 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" Oct 30 18:05:46.057332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.057199 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerDied","Data":"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2"} Oct 30 18:05:46.057332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.057258 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk" event={"ID":"f1677362-ae19-4863-9ddb-0ea0a6841a16","Type":"ContainerDied","Data":"bf89ef12c6b7936e06adc807a9d12e019adfcd427381133f1106493037ed488f"} Oct 30 18:05:46.057332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.057303 4622 scope.go:122] "RemoveContainer" containerID="b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2" Oct 30 18:05:46.068877 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.068858 4622 scope.go:122] "RemoveContainer" containerID="450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b" Oct 30 18:05:46.079203 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.079179 4622 scope.go:122] "RemoveContainer" containerID="b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2" Oct 30 18:05:46.079482 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:46.079464 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2\": container with ID starting with b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2 not found: ID does not exist" containerID="b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2" Oct 30 18:05:46.079552 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.079490 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2"} err="failed to get container status \"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2\": rpc error: code = NotFound desc = could not find container \"b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2\": container with ID starting with b829d4c9a10d57450a0a654a358907a2c59241f03a89d78af0bfab4e9ff456e2 not found: ID does not exist" Oct 30 18:05:46.079552 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.079506 4622 scope.go:122] "RemoveContainer" containerID="450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b" Oct 30 18:05:46.079735 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:46.079720 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b\": container with ID starting with 450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b not found: ID does not exist" containerID="450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b" Oct 30 18:05:46.079777 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.079738 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b"} err="failed to get container status \"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b\": rpc error: code = NotFound desc = could not find container \"450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b\": container with ID starting with 450d085773c4d3af9b12bf5bd11d2398d4249a88ecb074e4af70b536a6625f7b not found: ID does not exist" Oct 30 18:05:46.144134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.144097 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:05:46.147321 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.147300 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10dda-openshift-default-74ccddbffc-9g7tk"] Oct 30 18:05:46.374495 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:46.374426 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" path="/var/lib/kubelet/pods/f1677362-ae19-4863-9ddb-0ea0a6841a16/volumes" Oct 30 18:05:50.603285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:50.593695 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 18:05:50.603285 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:50.593944 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" podUID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" containerName="maas-api" containerID="cri-o://c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd" gracePeriod=30 Oct 30 18:05:51.002874 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.002846 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 18:05:51.059291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.058451 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 18:05:51.059291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.058690 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" podUID="10630542-7a98-4003-a343-2498ff3cdd99" containerName="maas-api" containerID="cri-o://eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f" gracePeriod=30 Oct 30 18:05:51.139318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.135817 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\") pod \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " Oct 30 18:05:51.139318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.135896 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\" (UniqueName: \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\") pod \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " Oct 30 18:05:51.139318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.135974 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\") pod \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " Oct 30 18:05:51.139318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.136020 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\") pod \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\" (UID: \"a65c9f56-db3d-4ab3-85d4-25f1c4f72684\") " Oct 30 18:05:51.139318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.137898 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca" pod "a65c9f56-db3d-4ab3-85d4-25f1c4f72684" (UID: "a65c9f56-db3d-4ab3-85d4-25f1c4f72684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:51.142772 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.140461 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs" pod "a65c9f56-db3d-4ab3-85d4-25f1c4f72684" (UID: "a65c9f56-db3d-4ab3-85d4-25f1c4f72684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:51.143119 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.143089 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls" pod "a65c9f56-db3d-4ab3-85d4-25f1c4f72684" (UID: "a65c9f56-db3d-4ab3-85d4-25f1c4f72684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:51.144637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.144211 4622 generic.go:356] "Generic (PLEG): container finished" podID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" containerID="c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd" exitCode=0 Oct 30 18:05:51.144637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.144297 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" event={"ID":"a65c9f56-db3d-4ab3-85d4-25f1c4f72684","Type":"ContainerDied","Data":"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd"} Oct 30 18:05:51.144637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.144357 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" event={"ID":"a65c9f56-db3d-4ab3-85d4-25f1c4f72684","Type":"ContainerDied","Data":"2a491d0b2e645f13c93e212735b2e86e705c19d17bf7f90aa2d33abde7d27262"} Oct 30 18:05:51.144637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.144380 4622 scope.go:122] "RemoveContainer" containerID="c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd" Oct 30 18:05:51.144943 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.144774 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl" Oct 30 18:05:51.148966 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.147622 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww" pod "a65c9f56-db3d-4ab3-85d4-25f1c4f72684" (UID: "a65c9f56-db3d-4ab3-85d4-25f1c4f72684"). InnerVolumeSpecName "kube-api-access-4xpww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:51.186435 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.186410 4622 scope.go:122] "RemoveContainer" containerID="c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd" Oct 30 18:05:51.186815 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:51.186779 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd\": container with ID starting with c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd not found: ID does not exist" containerID="c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd" Oct 30 18:05:51.186903 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.186818 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd"} err="failed to get container status \"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd\": rpc error: code = NotFound desc = could not find container \"c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd\": container with ID starting with c71cfb6f810db768dc12e6d6fc3d717eb4199dc7a5dd494274b0844a08fbadbd not found: ID does not exist" Oct 30 18:05:51.239401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.239312 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.239401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.239347 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.239401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.239362 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.239401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.239377 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpww\" (UniqueName: \"kubernetes.io/projected/a65c9f56-db3d-4ab3-85d4-25f1c4f72684-kube-api-access-4xpww\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.388430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.388390 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 18:05:51.493450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.493376 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 18:05:51.497608 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.497578 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-306aed-dffc5bd8-2j5tl"] Oct 30 18:05:51.544480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.544449 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\") pod \"10630542-7a98-4003-a343-2498ff3cdd99\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " Oct 30 18:05:51.544619 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.544545 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\") pod \"10630542-7a98-4003-a343-2498ff3cdd99\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " Oct 30 18:05:51.544701 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.544626 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\" (UniqueName: \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\") pod \"10630542-7a98-4003-a343-2498ff3cdd99\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " Oct 30 18:05:51.544769 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.544707 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\") pod \"10630542-7a98-4003-a343-2498ff3cdd99\" (UID: \"10630542-7a98-4003-a343-2498ff3cdd99\") " Oct 30 18:05:51.544880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.544856 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca" pod "10630542-7a98-4003-a343-2498ff3cdd99" (UID: "10630542-7a98-4003-a343-2498ff3cdd99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:05:51.545134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.545110 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10630542-7a98-4003-a343-2498ff3cdd99-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.546825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.546798 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls" pod "10630542-7a98-4003-a343-2498ff3cdd99" (UID: "10630542-7a98-4003-a343-2498ff3cdd99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:51.546825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.546821 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs" pod "10630542-7a98-4003-a343-2498ff3cdd99" (UID: "10630542-7a98-4003-a343-2498ff3cdd99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:05:51.546954 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.546893 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885" pod "10630542-7a98-4003-a343-2498ff3cdd99" (UID: "10630542-7a98-4003-a343-2498ff3cdd99"). InnerVolumeSpecName "kube-api-access-j8885". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:51.647077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.647044 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.647077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.647074 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8885\" (UniqueName: \"kubernetes.io/projected/10630542-7a98-4003-a343-2498ff3cdd99-kube-api-access-j8885\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:51.647077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:51.647084 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10630542-7a98-4003-a343-2498ff3cdd99-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:05:52.156314 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.156252 4622 generic.go:356] "Generic (PLEG): container finished" podID="10630542-7a98-4003-a343-2498ff3cdd99" containerID="eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f" exitCode=0 Oct 30 18:05:52.156501 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.156310 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" event={"ID":"10630542-7a98-4003-a343-2498ff3cdd99","Type":"ContainerDied","Data":"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f"} Oct 30 18:05:52.156501 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.156331 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" Oct 30 18:05:52.156501 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.156370 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr" event={"ID":"10630542-7a98-4003-a343-2498ff3cdd99","Type":"ContainerDied","Data":"33bce92a5065b0b6778dc00a67f765a40c57d5e52b736910ab166b1b3b17d90a"} Oct 30 18:05:52.156501 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.156393 4622 scope.go:122] "RemoveContainer" containerID="eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f" Oct 30 18:05:52.166604 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.166585 4622 scope.go:122] "RemoveContainer" containerID="eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f" Oct 30 18:05:52.166861 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:52.166843 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f\": container with ID starting with eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f not found: ID does not exist" containerID="eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f" Oct 30 18:05:52.166933 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.166868 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f"} err="failed to get container status \"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f\": rpc error: code = NotFound desc = could not find container \"eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f\": container with ID starting with eab0e38debea9084952758f0d6c5fe11b5fda3123941aa9db3213a7d4927e45f not found: ID does not exist" Oct 30 18:05:52.205955 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.205924 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 18:05:52.216582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.216546 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-d207f8-6cdcdf6448-cc7mr"] Oct 30 18:05:52.375399 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.375369 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10630542-7a98-4003-a343-2498ff3cdd99" path="/var/lib/kubelet/pods/10630542-7a98-4003-a343-2498ff3cdd99/volumes" Oct 30 18:05:52.375870 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:05:52.375857 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" path="/var/lib/kubelet/pods/a65c9f56-db3d-4ab3-85d4-25f1c4f72684/volumes" Oct 30 18:05:54.533828 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:05:54.533787 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:06:00.234528 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:00.234489 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 18:06:00.235077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:00.234744 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" containerID="cri-o://27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5" gracePeriod=30 Oct 30 18:06:02.276953 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:02.276916 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 18:06:02.277465 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:02.277109 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" containerID="cri-o://c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76" gracePeriod=30 Oct 30 18:06:04.051073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.051036 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 18:06:04.051575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.051252 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" podUID="c4987861-f6c8-4363-b616-14887a22b22d" containerName="maas-api" containerID="cri-o://81f1bb46fe6773601233c9274db09f2a5ae1194a32e2471cd3809c87aee79ce2" gracePeriod=30 Oct 30 18:06:04.249170 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.249036 4622 generic.go:356] "Generic (PLEG): container finished" podID="c4987861-f6c8-4363-b616-14887a22b22d" containerID="81f1bb46fe6773601233c9274db09f2a5ae1194a32e2471cd3809c87aee79ce2" exitCode=0 Oct 30 18:06:04.249170 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.249107 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" event={"ID":"c4987861-f6c8-4363-b616-14887a22b22d","Type":"ContainerDied","Data":"81f1bb46fe6773601233c9274db09f2a5ae1194a32e2471cd3809c87aee79ce2"} Oct 30 18:06:04.339564 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.333665 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 18:06:04.424160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.421938 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\") pod \"c4987861-f6c8-4363-b616-14887a22b22d\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " Oct 30 18:06:04.424160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.422034 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\") pod \"c4987861-f6c8-4363-b616-14887a22b22d\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " Oct 30 18:06:04.424160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.422077 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\") pod \"c4987861-f6c8-4363-b616-14887a22b22d\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " Oct 30 18:06:04.424160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.422124 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\" (UniqueName: \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\") pod \"c4987861-f6c8-4363-b616-14887a22b22d\" (UID: \"c4987861-f6c8-4363-b616-14887a22b22d\") " Oct 30 18:06:04.424160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.423000 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca" pod "c4987861-f6c8-4363-b616-14887a22b22d" (UID: "c4987861-f6c8-4363-b616-14887a22b22d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:04.424765 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.424741 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls" pod "c4987861-f6c8-4363-b616-14887a22b22d" (UID: "c4987861-f6c8-4363-b616-14887a22b22d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:04.425988 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.425956 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs" pod "c4987861-f6c8-4363-b616-14887a22b22d" (UID: "c4987861-f6c8-4363-b616-14887a22b22d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:04.427465 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.427441 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8" pod "c4987861-f6c8-4363-b616-14887a22b22d" (UID: "c4987861-f6c8-4363-b616-14887a22b22d"). InnerVolumeSpecName "kube-api-access-rbpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:04.525230 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.525185 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4987861-f6c8-4363-b616-14887a22b22d-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:04.525800 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.525777 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:04.525936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.525923 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4987861-f6c8-4363-b616-14887a22b22d-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:04.526024 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:04.526012 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbpw8\" (UniqueName: \"kubernetes.io/projected/c4987861-f6c8-4363-b616-14887a22b22d-kube-api-access-rbpw8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.259676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.258825 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" Oct 30 18:06:05.259676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.258832 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs" event={"ID":"c4987861-f6c8-4363-b616-14887a22b22d","Type":"ContainerDied","Data":"4bc52d884310247be782e3b9844bf17441bb2c10c7d50bcc9b018ee56ff95293"} Oct 30 18:06:05.259676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.258910 4622 scope.go:122] "RemoveContainer" containerID="81f1bb46fe6773601233c9274db09f2a5ae1194a32e2471cd3809c87aee79ce2" Oct 30 18:06:05.373943 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.373881 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 18:06:05.383465 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.383442 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-8d1062-5cfb598c5d-fzrxs"] Oct 30 18:06:05.517810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.517752 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 18:06:05.533961 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.533934 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.533992 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534016 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534042 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534057 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534102 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534176 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534199 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.534331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534243 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534306 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\") pod \"3de896bf-fccf-4677-9b62-97038fc12bca\" (UID: \"3de896bf-fccf-4677-9b62-97038fc12bca\") " Oct 30 18:06:05.534619 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534340 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.534619 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534425 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.534619 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534545 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.534803 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534707 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:05.534857 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534839 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.534912 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534857 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.534912 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534868 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.534912 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534880 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.534912 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.534891 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3de896bf-fccf-4677-9b62-97038fc12bca-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.536324 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.536300 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:05.536435 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.536319 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:05.536677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.536661 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:05.536758 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.536687 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h" pod "3de896bf-fccf-4677-9b62-97038fc12bca" (UID: "3de896bf-fccf-4677-9b62-97038fc12bca"). InnerVolumeSpecName "kube-api-access-nbd2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:05.635785 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.635752 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3de896bf-fccf-4677-9b62-97038fc12bca-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.635785 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.635786 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3de896bf-fccf-4677-9b62-97038fc12bca-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.635961 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.635800 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbd2h\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-kube-api-access-nbd2h\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:05.635961 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:05.635814 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3de896bf-fccf-4677-9b62-97038fc12bca-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:06.267923 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.267882 4622 generic.go:356] "Generic (PLEG): container finished" podID="3de896bf-fccf-4677-9b62-97038fc12bca" containerID="27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5" exitCode=0 Oct 30 18:06:06.268405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.267945 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" Oct 30 18:06:06.268405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.267968 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" event={"ID":"3de896bf-fccf-4677-9b62-97038fc12bca","Type":"ContainerDied","Data":"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5"} Oct 30 18:06:06.268405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.268019 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s" event={"ID":"3de896bf-fccf-4677-9b62-97038fc12bca","Type":"ContainerDied","Data":"892ea75d5da5f352d5dd614aeb285eb5444dbbee3784625c24d62be460d694e6"} Oct 30 18:06:06.268405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.268040 4622 scope.go:122] "RemoveContainer" containerID="27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5" Oct 30 18:06:06.278693 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.278677 4622 scope.go:122] "RemoveContainer" containerID="27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5" Oct 30 18:06:06.278945 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:06.278927 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5\": container with ID starting with 27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5 not found: ID does not exist" containerID="27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5" Oct 30 18:06:06.278994 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.278953 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5"} err="failed to get container status \"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5\": rpc error: code = NotFound desc = could not find container \"27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5\": container with ID starting with 27f37e8756171498d35247867abe685ee1d29d9dcb4bf78e45d8126d9350a7c5 not found: ID does not exist" Oct 30 18:06:06.340055 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.340029 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 18:06:06.342921 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.342892 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-d207f8-openshift-default-b8cdf9f49-ljw8s"] Oct 30 18:06:06.378374 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.378341 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" path="/var/lib/kubelet/pods/3de896bf-fccf-4677-9b62-97038fc12bca/volumes" Oct 30 18:06:06.378802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:06.378786 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4987861-f6c8-4363-b616-14887a22b22d" path="/var/lib/kubelet/pods/c4987861-f6c8-4363-b616-14887a22b22d/volumes" Oct 30 18:06:07.435893 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.435858 4622 patch_prober.go:29] interesting pod/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:06:07.436355 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.435932 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:06:07.548092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.548019 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 18:06:07.556507 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556479 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556574 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556683 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556648 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556735 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556706 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556755 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556767 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:07.556797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556787 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556820 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556857 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.556960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.556937 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\") pod \"a7e53580-8fc3-452d-ad54-58b3ba82215a\" (UID: \"a7e53580-8fc3-452d-ad54-58b3ba82215a\") " Oct 30 18:06:07.557133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557011 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:07.557133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557026 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:07.557133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557034 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:07.557133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557128 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:07.557410 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557389 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.557410 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557409 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.557594 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557419 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.557594 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557428 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.557594 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.557436 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7e53580-8fc3-452d-ad54-58b3ba82215a-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.558967 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.558940 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:07.559076 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.559016 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "kube-api-access-htsg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:07.559143 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.559086 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:07.559203 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.559161 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy" pod "a7e53580-8fc3-452d-ad54-58b3ba82215a" (UID: "a7e53580-8fc3-452d-ad54-58b3ba82215a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:07.658908 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.658878 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htsg2\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-kube-api-access-htsg2\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.658908 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.658906 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.659080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.658916 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:07.659080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:07.658925 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7e53580-8fc3-452d-ad54-58b3ba82215a-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:08.283305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.283272 4622 generic.go:356] "Generic (PLEG): container finished" podID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerID="c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76" exitCode=0 Oct 30 18:06:08.283505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.283334 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" Oct 30 18:06:08.283505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.283352 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" event={"ID":"a7e53580-8fc3-452d-ad54-58b3ba82215a","Type":"ContainerDied","Data":"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76"} Oct 30 18:06:08.283505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.283429 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v" event={"ID":"a7e53580-8fc3-452d-ad54-58b3ba82215a","Type":"ContainerDied","Data":"9d867fa4190235b1fd040cb787f0c39141c8f2b33c2fce71944d4a9a3aa60bad"} Oct 30 18:06:08.283505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.283453 4622 scope.go:122] "RemoveContainer" containerID="c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76" Oct 30 18:06:08.293907 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.293885 4622 scope.go:122] "RemoveContainer" containerID="c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76" Oct 30 18:06:08.294158 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:08.294138 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76\": container with ID starting with c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76 not found: ID does not exist" containerID="c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76" Oct 30 18:06:08.294199 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.294167 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76"} err="failed to get container status \"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76\": rpc error: code = NotFound desc = could not find container \"c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76\": container with ID starting with c43d25c6c2ae666161e62437f82ba17eb02a0c86a991c6f76ca798fee9740a76 not found: ID does not exist" Oct 30 18:06:08.342772 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.342744 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 18:06:08.353219 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.353180 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-d53ec1-openshift-default-9dd47fff-92k9v"] Oct 30 18:06:08.374577 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:08.374556 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" path="/var/lib/kubelet/pods/a7e53580-8fc3-452d-ad54-58b3ba82215a/volumes" Oct 30 18:06:11.428701 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:11.428664 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 18:06:16.429513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:16.429476 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" containerID="cri-o://55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4" gracePeriod=25 Oct 30 18:06:22.324982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:22.324944 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:24.533695 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:24.533658 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:06:29.294052 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.294011 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:06:29.294557 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.294300 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" podUID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" containerName="maas-api" containerID="cri-o://82f4b68097424e114202048c693766124db2da6b247ad44beb6de22ced7289a9" gracePeriod=30 Oct 30 18:06:29.430759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.430714 4622 generic.go:356] "Generic (PLEG): container finished" podID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" containerID="82f4b68097424e114202048c693766124db2da6b247ad44beb6de22ced7289a9" exitCode=0 Oct 30 18:06:29.430915 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.430772 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" event={"ID":"830c8d6a-6b64-4ec5-8eea-fce1f067a883","Type":"ContainerDied","Data":"82f4b68097424e114202048c693766124db2da6b247ad44beb6de22ced7289a9"} Oct 30 18:06:29.631162 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.631132 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:06:29.730160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.730123 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\") pod \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " Oct 30 18:06:29.730367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.730351 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\") pod \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " Oct 30 18:06:29.730424 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.730410 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\") pod \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " Oct 30 18:06:29.730556 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.730526 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\" (UniqueName: \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\") pod \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\" (UID: \"830c8d6a-6b64-4ec5-8eea-fce1f067a883\") " Oct 30 18:06:29.730724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.730705 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca" pod "830c8d6a-6b64-4ec5-8eea-fce1f067a883" (UID: "830c8d6a-6b64-4ec5-8eea-fce1f067a883"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:29.731022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.731000 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/830c8d6a-6b64-4ec5-8eea-fce1f067a883-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:29.732287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.732245 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs" pod "830c8d6a-6b64-4ec5-8eea-fce1f067a883" (UID: "830c8d6a-6b64-4ec5-8eea-fce1f067a883"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:29.732547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.732530 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls" pod "830c8d6a-6b64-4ec5-8eea-fce1f067a883" (UID: "830c8d6a-6b64-4ec5-8eea-fce1f067a883"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:29.732618 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.732573 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2" pod "830c8d6a-6b64-4ec5-8eea-fce1f067a883" (UID: "830c8d6a-6b64-4ec5-8eea-fce1f067a883"). InnerVolumeSpecName "kube-api-access-bkgn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:29.832368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.832322 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:29.832368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.832357 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgn2\" (UniqueName: \"kubernetes.io/projected/830c8d6a-6b64-4ec5-8eea-fce1f067a883-kube-api-access-bkgn2\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:29.832368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:29.832373 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/830c8d6a-6b64-4ec5-8eea-fce1f067a883-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:30.438543 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:30.438504 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" event={"ID":"830c8d6a-6b64-4ec5-8eea-fce1f067a883","Type":"ContainerDied","Data":"75b0388632ddea19613363ec10c34ca40508f0969cccdddc28ad627b1f094dd9"} Oct 30 18:06:30.438901 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:30.438560 4622 scope.go:122] "RemoveContainer" containerID="82f4b68097424e114202048c693766124db2da6b247ad44beb6de22ced7289a9" Oct 30 18:06:30.438901 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:30.438518 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp" Oct 30 18:06:30.488113 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:30.488082 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:06:30.498763 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:30.498741 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584a12-6468f97bc6-hpnkp"] Oct 30 18:06:31.665847 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:31.665797 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 18:06:31.666296 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:31.666062 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" podUID="150533ae-1f35-493b-bc3c-525065322660" containerName="maas-api" containerID="cri-o://a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94" gracePeriod=30 Oct 30 18:06:31.974556 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:31.974518 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 18:06:32.070864 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.070823 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\") pod \"150533ae-1f35-493b-bc3c-525065322660\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " Oct 30 18:06:32.071075 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.070915 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\" (UniqueName: \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\") pod \"150533ae-1f35-493b-bc3c-525065322660\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " Oct 30 18:06:32.071075 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.070939 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\") pod \"150533ae-1f35-493b-bc3c-525065322660\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " Oct 30 18:06:32.071075 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.070967 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k"] Oct 30 18:06:32.071075 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.071049 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\") pod \"150533ae-1f35-493b-bc3c-525065322660\" (UID: \"150533ae-1f35-493b-bc3c-525065322660\") " Oct 30 18:06:32.072255 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072229 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" containerName="maas-api" Oct 30 18:06:32.072255 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072253 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" containerName="maas-api" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072297 4622 cpu_manager.go:414] "containerMap: removing container" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072308 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072337 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072352 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072373 4622 cpu_manager.go:414] "containerMap: removing container" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072383 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072394 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072402 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072413 4622 cpu_manager.go:414] "containerMap: removing container" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072428 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072440 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072447 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072465 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" Oct 30 18:06:32.072484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072473 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072526 4622 cpu_manager.go:414] "containerMap: removing container" podUID="150533ae-1f35-493b-bc3c-525065322660" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072535 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150533ae-1f35-493b-bc3c-525065322660" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072554 4622 cpu_manager.go:414] "containerMap: removing container" podUID="116cf21b-2633-4a7c-aebf-3d7855282cce" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072562 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116cf21b-2633-4a7c-aebf-3d7855282cce" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072574 4622 cpu_manager.go:414] "containerMap: removing container" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072580 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072598 4622 cpu_manager.go:414] "containerMap: removing container" podUID="59a85f43-178e-4a63-9712-30ff98dd8d43" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072605 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a85f43-178e-4a63-9712-30ff98dd8d43" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072629 4622 cpu_manager.go:414] "containerMap: removing container" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072636 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072655 4622 cpu_manager.go:414] "containerMap: removing container" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072663 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072677 4622 cpu_manager.go:414] "containerMap: removing container" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072684 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072701 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072709 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072721 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072728 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072760 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c4987861-f6c8-4363-b616-14887a22b22d" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072770 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4987861-f6c8-4363-b616-14887a22b22d" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072796 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072809 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072836 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072892 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072925 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072935 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072954 4622 cpu_manager.go:414] "containerMap: removing container" podUID="01a94584-506d-4b6d-81d8-88e8ddd408d4" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072962 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a94584-506d-4b6d-81d8-88e8ddd408d4" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072990 4622 cpu_manager.go:414] "containerMap: removing container" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.072997 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073018 4622 cpu_manager.go:414] "containerMap: removing container" podUID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073025 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073057 4622 cpu_manager.go:414] "containerMap: removing container" podUID="10630542-7a98-4003-a343-2498ff3cdd99" containerName="maas-api" Oct 30 18:06:32.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073065 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10630542-7a98-4003-a343-2498ff3cdd99" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073317 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b362a8b-9719-4a90-ba33-81417c635064" containerName="payload-pre-processing" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073342 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e53580-8fc3-452d-ad54-58b3ba82215a" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073358 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073369 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073376 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd1506e-d8a3-432d-8180-60377fe3c5a1" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073390 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073398 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c80672-f796-4050-bcfe-c274eb4c805d" containerName="main" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073420 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="150533ae-1f35-493b-bc3c-525065322660" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073444 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="116cf21b-2633-4a7c-aebf-3d7855282cce" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073462 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de896bf-fccf-4677-9b62-97038fc12bca" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073509 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a85f43-178e-4a63-9712-30ff98dd8d43" containerName="main" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073523 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="465185c0-eef3-44d5-9b65-ec78a387a45d" containerName="main" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073541 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073558 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073583 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073593 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aec6088-65d8-406f-a00f-3c51adfc400a" containerName="main" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073611 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4987861-f6c8-4363-b616-14887a22b22d" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073622 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a615c998-26e0-49ed-8878-6b1c0f7b1d9e" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073641 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1677362-ae19-4863-9ddb-0ea0a6841a16" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073678 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2323cb-9feb-4803-92f1-8f2146ebdc43" containerName="istio-proxy" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073696 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a94584-506d-4b6d-81d8-88e8ddd408d4" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073713 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c442a99-d65a-49df-a1cf-79938df4f4ac" containerName="main" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073724 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073766 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="10630542-7a98-4003-a343-2498ff3cdd99" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.073778 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65c9f56-db3d-4ab3-85d4-25f1c4f72684" containerName="maas-api" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.078532 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca" pod "150533ae-1f35-493b-bc3c-525065322660" (UID: "150533ae-1f35-493b-bc3c-525065322660"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:32.091751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.087059 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls" pod "150533ae-1f35-493b-bc3c-525065322660" (UID: "150533ae-1f35-493b-bc3c-525065322660"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:32.096395 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.096341 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg" pod "150533ae-1f35-493b-bc3c-525065322660" (UID: "150533ae-1f35-493b-bc3c-525065322660"). InnerVolumeSpecName "kube-api-access-mgvgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:32.098867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.096794 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k"] Oct 30 18:06:32.098867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.096934 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.100658 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.100630 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs" pod "150533ae-1f35-493b-bc3c-525065322660" (UID: "150533ae-1f35-493b-bc3c-525065322660"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:32.110582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.110554 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:32.110742 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.110723 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:06:32.172626 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.172592 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:32.172626 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.172619 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/150533ae-1f35-493b-bc3c-525065322660-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:32.172626 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.172628 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvgg\" (UniqueName: \"kubernetes.io/projected/150533ae-1f35-493b-bc3c-525065322660-kube-api-access-mgvgg\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:32.172853 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.172638 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/150533ae-1f35-493b-bc3c-525065322660-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:32.274308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.274208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.274308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.274252 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.274544 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.274468 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7mk\" (UniqueName: \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.324358 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.324324 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:06:32.374078 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.374053 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830c8d6a-6b64-4ec5-8eea-fce1f067a883" path="/var/lib/kubelet/pods/830c8d6a-6b64-4ec5-8eea-fce1f067a883/volumes" Oct 30 18:06:32.375330 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.375305 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7mk\" (UniqueName: \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.375425 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.375410 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.375469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.375446 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.376239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.376215 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.377775 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.377754 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.396715 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.396690 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7mk\" (UniqueName: \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\") pod \"maas-api-revoke-sub-90683a54-zbl7k\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.421571 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.421552 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:32.454520 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.454481 4622 generic.go:356] "Generic (PLEG): container finished" podID="150533ae-1f35-493b-bc3c-525065322660" containerID="a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94" exitCode=0 Oct 30 18:06:32.454663 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.454534 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" Oct 30 18:06:32.454663 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.454560 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" event={"ID":"150533ae-1f35-493b-bc3c-525065322660","Type":"ContainerDied","Data":"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94"} Oct 30 18:06:32.454663 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.454609 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn" event={"ID":"150533ae-1f35-493b-bc3c-525065322660","Type":"ContainerDied","Data":"7bbc1e366bf3fc4e0115df20921e76ae5e220a1e074c602f3a5f393dcf9f2534"} Oct 30 18:06:32.454663 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.454635 4622 scope.go:122] "RemoveContainer" containerID="a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94" Oct 30 18:06:32.465000 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.464980 4622 scope.go:122] "RemoveContainer" containerID="a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94" Oct 30 18:06:32.465306 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:32.465282 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94\": container with ID starting with a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94 not found: ID does not exist" containerID="a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94" Oct 30 18:06:32.465399 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.465313 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94"} err="failed to get container status \"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94\": rpc error: code = NotFound desc = could not find container \"a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94\": container with ID starting with a54c3fe79c5d868d9172927793a8590004910cdeb678010e4bd96a2918fd7a94 not found: ID does not exist" Oct 30 18:06:32.518216 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.518177 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 18:06:32.526970 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.526884 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-4dcdbb-66b477c849-jbqmn"] Oct 30 18:06:32.717730 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:32.717701 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k"] Oct 30 18:06:32.720127 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:06:32.720091 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4f2cac_7bca_4af6_8137_721d7dff2276.slice/crio-17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630 WatchSource:0}: Error finding container 17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630: Status 404 returned error can't find the container with id 17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630 Oct 30 18:06:33.463763 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.463720 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" event={"ID":"1a4f2cac-7bca-4af6-8137-721d7dff2276","Type":"ContainerStarted","Data":"8a9df2be3a07b73952841cf68577c615a32b661ae9ff8f51ba833efb2ed1cab2"} Oct 30 18:06:33.463949 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.463788 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" event={"ID":"1a4f2cac-7bca-4af6-8137-721d7dff2276","Type":"ContainerStarted","Data":"17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630"} Oct 30 18:06:33.532508 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.532447 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" podStartSLOduration=1.532430997 podStartE2EDuration="1.532430997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:33.530529834 +0000 UTC m=+3221.308001789" watchObservedRunningTime="2026-10-30 18:06:33.532430997 +0000 UTC m=+3221.309902953" Oct 30 18:06:33.910798 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.910765 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 18:06:33.915651 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.915622 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 18:06:33.938344 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:33.938317 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 18:06:34.374243 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:34.374210 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150533ae-1f35-493b-bc3c-525065322660" path="/var/lib/kubelet/pods/150533ae-1f35-493b-bc3c-525065322660/volumes" Oct 30 18:06:34.471881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:34.471845 4622 generic.go:356] "Generic (PLEG): container finished" podID="1a4f2cac-7bca-4af6-8137-721d7dff2276" containerID="8a9df2be3a07b73952841cf68577c615a32b661ae9ff8f51ba833efb2ed1cab2" exitCode=0 Oct 30 18:06:34.472064 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:34.471926 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" event={"ID":"1a4f2cac-7bca-4af6-8137-721d7dff2276","Type":"ContainerDied","Data":"8a9df2be3a07b73952841cf68577c615a32b661ae9ff8f51ba833efb2ed1cab2"} Oct 30 18:06:35.629698 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.629664 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:35.815568 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.815539 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\") pod \"1a4f2cac-7bca-4af6-8137-721d7dff2276\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " Oct 30 18:06:35.815730 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.815679 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\" (UniqueName: \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\") pod \"1a4f2cac-7bca-4af6-8137-721d7dff2276\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " Oct 30 18:06:35.815791 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.815765 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\") pod \"1a4f2cac-7bca-4af6-8137-721d7dff2276\" (UID: \"1a4f2cac-7bca-4af6-8137-721d7dff2276\") " Oct 30 18:06:35.815959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.815929 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca" pod "1a4f2cac-7bca-4af6-8137-721d7dff2276" (UID: "1a4f2cac-7bca-4af6-8137-721d7dff2276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:35.816158 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.816142 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.817706 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.817682 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca" pod "1a4f2cac-7bca-4af6-8137-721d7dff2276" (UID: "1a4f2cac-7bca-4af6-8137-721d7dff2276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:35.817809 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.817709 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk" pod "1a4f2cac-7bca-4af6-8137-721d7dff2276" (UID: "1a4f2cac-7bca-4af6-8137-721d7dff2276"). InnerVolumeSpecName "kube-api-access-9r7mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:35.917777 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.917746 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4f2cac-7bca-4af6-8137-721d7dff2276-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:35.917777 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:35.917774 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7mk\" (UniqueName: \"kubernetes.io/projected/1a4f2cac-7bca-4af6-8137-721d7dff2276-kube-api-access-9r7mk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:36.486838 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.486809 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" Oct 30 18:06:36.486972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.486816 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k" event={"ID":"1a4f2cac-7bca-4af6-8137-721d7dff2276","Type":"ContainerDied","Data":"17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630"} Oct 30 18:06:36.486972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.486936 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c03d4be7dbf6069d8319fff78d8a71687839f566405f7555af5921cbe89630" Oct 30 18:06:36.690082 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.690055 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 18:06:36.827012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.826974 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\") pod \"998d5375-3e23-44a4-83a3-b97150984c71\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " Oct 30 18:06:36.827214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.827191 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\" (UniqueName: \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\") pod \"998d5375-3e23-44a4-83a3-b97150984c71\" (UID: \"998d5375-3e23-44a4-83a3-b97150984c71\") " Oct 30 18:06:36.827414 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.827391 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume" pod "998d5375-3e23-44a4-83a3-b97150984c71" (UID: "998d5375-3e23-44a4-83a3-b97150984c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:36.827694 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.827670 4622 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998d5375-3e23-44a4-83a3-b97150984c71-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:36.829342 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.829316 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj" pod "998d5375-3e23-44a4-83a3-b97150984c71" (UID: "998d5375-3e23-44a4-83a3-b97150984c71"). InnerVolumeSpecName "kube-api-access-czzjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:36.928605 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:36.928539 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czzjj\" (UniqueName: \"kubernetes.io/projected/998d5375-3e23-44a4-83a3-b97150984c71-kube-api-access-czzjj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:37.495244 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.495208 4622 generic.go:356] "Generic (PLEG): container finished" podID="998d5375-3e23-44a4-83a3-b97150984c71" containerID="55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4" exitCode=0 Oct 30 18:06:37.495441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.495255 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" Oct 30 18:06:37.495441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.495290 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" event={"ID":"998d5375-3e23-44a4-83a3-b97150984c71","Type":"ContainerDied","Data":"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4"} Oct 30 18:06:37.495441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.495347 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4" event={"ID":"998d5375-3e23-44a4-83a3-b97150984c71","Type":"ContainerDied","Data":"0d04d5fb8f1c8fcb9b14743ee030e2c7a13f7851175a1c4507dcc03fb98674da"} Oct 30 18:06:37.495441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.495376 4622 scope.go:122] "RemoveContainer" containerID="55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4" Oct 30 18:06:37.507744 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.507720 4622 scope.go:122] "RemoveContainer" containerID="55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4" Oct 30 18:06:37.508040 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:37.508017 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4\": container with ID starting with 55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4 not found: ID does not exist" containerID="55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4" Oct 30 18:06:37.508139 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.508049 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4"} err="failed to get container status \"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4\": rpc error: code = NotFound desc = could not find container \"55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4\": container with ID starting with 55ab3c64a3989103f0003f9612ec5c3c8fd935eff23d57785401a5a547c994d4 not found: ID does not exist" Oct 30 18:06:37.542602 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.542580 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 18:06:37.552855 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:37.552830 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-db98b0-7cc945bcfd-w9dg4"] Oct 30 18:06:38.374351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:38.374316 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998d5375-3e23-44a4-83a3-b97150984c71" path="/var/lib/kubelet/pods/998d5375-3e23-44a4-83a3-b97150984c71/volumes" Oct 30 18:06:39.646871 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:39.646835 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 18:06:39.647404 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:39.647077 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" containerID="cri-o://d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2" gracePeriod=30 Oct 30 18:06:44.906356 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:44.906330 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 18:06:45.029059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.028978 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029017 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029047 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029081 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029137 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029187 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029412 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029358 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029424 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029496 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\") pod \"56c400d2-e231-4269-bced-1a5235ae7cf2\" (UID: \"56c400d2-e231-4269-bced-1a5235ae7cf2\") " Oct 30 18:06:45.029657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029353 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:45.029657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029426 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:45.029657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029558 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:45.029909 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029800 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:45.029909 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.029735 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:45.030032 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.030008 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56c400d2-e231-4269-bced-1a5235ae7cf2-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.030175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.030036 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.030175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.030053 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.030175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.030066 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.030175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.030080 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.032242 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.032216 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:45.035012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.034987 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:45.035140 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.035021 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "kube-api-access-8gbtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:45.035140 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.035039 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token" pod "56c400d2-e231-4269-bced-1a5235ae7cf2" (UID: "56c400d2-e231-4269-bced-1a5235ae7cf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:45.131121 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.131086 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.131121 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.131116 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gbtt\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-kube-api-access-8gbtt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.131121 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.131127 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.131380 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.131137 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56c400d2-e231-4269-bced-1a5235ae7cf2-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:45.548581 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.548544 4622 generic.go:356] "Generic (PLEG): container finished" podID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerID="d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2" exitCode=0 Oct 30 18:06:45.548764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.548601 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" Oct 30 18:06:45.548764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.548631 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerDied","Data":"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2"} Oct 30 18:06:45.548764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.548702 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76" event={"ID":"56c400d2-e231-4269-bced-1a5235ae7cf2","Type":"ContainerDied","Data":"76f5d3d77013de8e3bfa9c38c4843ead279b25bb0288540e059759c4f6541909"} Oct 30 18:06:45.548764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.548733 4622 scope.go:122] "RemoveContainer" containerID="d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2" Oct 30 18:06:45.561138 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.561119 4622 scope.go:122] "RemoveContainer" containerID="018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed" Oct 30 18:06:45.572992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.572966 4622 scope.go:122] "RemoveContainer" containerID="d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2" Oct 30 18:06:45.573564 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:45.573535 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2\": container with ID starting with d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2 not found: ID does not exist" containerID="d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2" Oct 30 18:06:45.573681 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.573573 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2"} err="failed to get container status \"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2\": rpc error: code = NotFound desc = could not find container \"d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2\": container with ID starting with d3e727f58d5ae2763186482c24127b4ef10e9438730a5017923ade8a368abdf2 not found: ID does not exist" Oct 30 18:06:45.573681 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.573600 4622 scope.go:122] "RemoveContainer" containerID="018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed" Oct 30 18:06:45.574055 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:45.574032 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed\": container with ID starting with 018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed not found: ID does not exist" containerID="018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed" Oct 30 18:06:45.574149 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.574062 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed"} err="failed to get container status \"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed\": rpc error: code = NotFound desc = could not find container \"018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed\": container with ID starting with 018d0cbda064d26d4b00d0f964161071fdbcdcff49952a332a1a084c48676fed not found: ID does not exist" Oct 30 18:06:45.604385 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.604357 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 18:06:45.614583 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:45.614397 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-4dcdbb-openshift-default-f766b48f5-66h76"] Oct 30 18:06:46.373814 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:46.373781 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" path="/var/lib/kubelet/pods/56c400d2-e231-4269-bced-1a5235ae7cf2/volumes" Oct 30 18:06:47.484866 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:47.484828 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:06:47.485612 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:47.485553 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" containerID="cri-o://2328794aea90130b4dc484873ffae9b702bae7eebde70f5d9310994907b74027" gracePeriod=30 Oct 30 18:06:48.935610 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:48.935570 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" containerID="cri-o://d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559" gracePeriod=45 Oct 30 18:06:48.940516 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:48.940486 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" containerID="cri-o://5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4" gracePeriod=45 Oct 30 18:06:48.958622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:48.958587 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" containerID="cri-o://4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef" gracePeriod=45 Oct 30 18:06:48.972043 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:48.972017 4622 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c06b9a2_2375_4173_afd0_7049113c75e5.slice/crio-d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:06:48.974784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:48.974752 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused" Oct 30 18:06:49.259151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.259121 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 18:06:49.281995 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.281917 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 18:06:49.290397 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.290377 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 18:06:49.375939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.375902 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.375959 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\" (UniqueName: \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.375990 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376026 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\" (UniqueName: \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376067 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376108 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\" (UniqueName: \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.376137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376133 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376167 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376198 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376216 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376237 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376281 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\" (UniqueName: \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376347 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376399 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\") pod \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\" (UID: \"3427dc4e-2da1-48d2-bfe3-859ab0d20550\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376439 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.376490 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376465 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.377203 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376526 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\" (UniqueName: \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\") pod \"559092cc-213f-45b7-aeb6-6aa816959162\" (UID: \"559092cc-213f-45b7-aeb6-6aa816959162\") " Oct 30 18:06:49.377203 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.376562 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\") pod \"5c06b9a2-2375-4173-afd0-7049113c75e5\" (UID: \"5c06b9a2-2375-4173-afd0-7049113c75e5\") " Oct 30 18:06:49.377528 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.377498 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.377724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.377697 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.377895 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.377759 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.377994 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.377930 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.378150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.378132 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.378296 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.378146 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.378678 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.378654 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.379006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.378985 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.379218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.379180 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.380430 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.380395 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.381894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.381863 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.382808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.382774 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:49.382969 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.382942 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "kube-api-access-kq4cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:49.383061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.383038 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:49.383178 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.383158 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs" pod "5c06b9a2-2375-4173-afd0-7049113c75e5" (UID: "5c06b9a2-2375-4173-afd0-7049113c75e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:49.383292 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.383251 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj" pod "559092cc-213f-45b7-aeb6-6aa816959162" (UID: "559092cc-213f-45b7-aeb6-6aa816959162"). InnerVolumeSpecName "kube-api-access-2t8kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:49.383725 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.383703 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "kube-api-access-vbbvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:49.384638 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.384612 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm" pod "3427dc4e-2da1-48d2-bfe3-859ab0d20550" (UID: "3427dc4e-2da1-48d2-bfe3-859ab0d20550"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478032 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478064 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478074 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478082 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478091 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8kj\" (UniqueName: \"kubernetes.io/projected/559092cc-213f-45b7-aeb6-6aa816959162-kube-api-access-2t8kj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478098 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478107 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq4cl\" (UniqueName: \"kubernetes.io/projected/5c06b9a2-2375-4173-afd0-7049113c75e5-kube-api-access-kq4cl\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478119 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478130 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478138 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478145 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478152 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559092cc-213f-45b7-aeb6-6aa816959162-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478160 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbvc\" (UniqueName: \"kubernetes.io/projected/3427dc4e-2da1-48d2-bfe3-859ab0d20550-kube-api-access-vbbvc\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478167 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478175 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3427dc4e-2da1-48d2-bfe3-859ab0d20550-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478182 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c06b9a2-2375-4173-afd0-7049113c75e5-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478191 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c06b9a2-2375-4173-afd0-7049113c75e5-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.478532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.478201 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559092cc-213f-45b7-aeb6-6aa816959162-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:49.576486 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.576453 4622 generic.go:356] "Generic (PLEG): container finished" podID="559092cc-213f-45b7-aeb6-6aa816959162" containerID="4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef" exitCode=0 Oct 30 18:06:49.576662 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.576495 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" Oct 30 18:06:49.576662 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.576530 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" event={"ID":"559092cc-213f-45b7-aeb6-6aa816959162","Type":"ContainerDied","Data":"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef"} Oct 30 18:06:49.576662 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.576575 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf" event={"ID":"559092cc-213f-45b7-aeb6-6aa816959162","Type":"ContainerDied","Data":"2eda88d3a1f81d91ed33c5202d89ef233d544b50c461e4a3fd5aa07ba96f6455"} Oct 30 18:06:49.576662 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.576598 4622 scope.go:122] "RemoveContainer" containerID="4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef" Oct 30 18:06:49.578004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.577982 4622 generic.go:356] "Generic (PLEG): container finished" podID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerID="5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4" exitCode=0 Oct 30 18:06:49.578117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.578014 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" event={"ID":"3427dc4e-2da1-48d2-bfe3-859ab0d20550","Type":"ContainerDied","Data":"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4"} Oct 30 18:06:49.578117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.578039 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" Oct 30 18:06:49.578117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.578051 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8" event={"ID":"3427dc4e-2da1-48d2-bfe3-859ab0d20550","Type":"ContainerDied","Data":"f476b63c092f67953bfb471a5dfebc0aebb767d1193c01cd94d4f27034f2a190"} Oct 30 18:06:49.579992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.579969 4622 generic.go:356] "Generic (PLEG): container finished" podID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerID="d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559" exitCode=0 Oct 30 18:06:49.580129 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.580016 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" Oct 30 18:06:49.580129 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.580063 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" event={"ID":"5c06b9a2-2375-4173-afd0-7049113c75e5","Type":"ContainerDied","Data":"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559"} Oct 30 18:06:49.580129 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.580088 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd" event={"ID":"5c06b9a2-2375-4173-afd0-7049113c75e5","Type":"ContainerDied","Data":"90a2f91cc654be7ff5840d71141a8e1d04263e49796e0f40b5a41bdd1a2eab72"} Oct 30 18:06:49.587286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.587252 4622 scope.go:122] "RemoveContainer" containerID="4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef" Oct 30 18:06:49.587626 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:49.587595 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef\": container with ID starting with 4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef not found: ID does not exist" containerID="4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef" Oct 30 18:06:49.587749 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.587635 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef"} err="failed to get container status \"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef\": rpc error: code = NotFound desc = could not find container \"4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef\": container with ID starting with 4a78e2119f47d297fdc62b9c59489f55fbdd1f70d66b5bd00bfde6251c4ecbef not found: ID does not exist" Oct 30 18:06:49.587749 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.587661 4622 scope.go:122] "RemoveContainer" containerID="5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4" Oct 30 18:06:49.597050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.597031 4622 scope.go:122] "RemoveContainer" containerID="5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4" Oct 30 18:06:49.597323 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:49.597297 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4\": container with ID starting with 5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4 not found: ID does not exist" containerID="5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4" Oct 30 18:06:49.597439 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.597334 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4"} err="failed to get container status \"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4\": rpc error: code = NotFound desc = could not find container \"5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4\": container with ID starting with 5679e8111fdf19bfae1aaac2b85d0428952ad64593dcb311687078fe882a7aa4 not found: ID does not exist" Oct 30 18:06:49.597439 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.597356 4622 scope.go:122] "RemoveContainer" containerID="d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559" Oct 30 18:06:49.605179 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.605165 4622 scope.go:122] "RemoveContainer" containerID="d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559" Oct 30 18:06:49.605429 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:49.605411 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559\": container with ID starting with d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559 not found: ID does not exist" containerID="d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559" Oct 30 18:06:49.605483 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.605434 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559"} err="failed to get container status \"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559\": rpc error: code = NotFound desc = could not find container \"d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559\": container with ID starting with d73ef08dadf17d40f2d9eb37fef8865076fc3e7093f0bc270ed02aa5fd35a559 not found: ID does not exist" Oct 30 18:06:49.657804 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.657780 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 18:06:49.662144 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.662121 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-506124/simulator-w2-506124-kserve-5ffc84887f-whpsf"] Oct 30 18:06:49.692125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.692098 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 18:06:49.699029 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.699007 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-w2-506124-kserve-54459649df-dcxs8"] Oct 30 18:06:49.728593 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.728536 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 18:06:49.734436 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.734410 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-506124/distinct-2-w2-506124-kserve-6c7fbb5f4b-prrcd"] Oct 30 18:06:49.787401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787369 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh"] Oct 30 18:06:49.787818 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787805 4622 cpu_manager.go:414] "containerMap: removing container" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787819 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787834 4622 cpu_manager.go:414] "containerMap: removing container" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787839 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787845 4622 cpu_manager.go:414] "containerMap: removing container" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787850 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787864 4622 cpu_manager.go:414] "containerMap: removing container" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787869 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:49.787875 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787877 4622 cpu_manager.go:414] "containerMap: removing container" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787882 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787904 4622 cpu_manager.go:414] "containerMap: removing container" podUID="1a4f2cac-7bca-4af6-8137-721d7dff2276" containerName="revoke-keys" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787909 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4f2cac-7bca-4af6-8137-721d7dff2276" containerName="revoke-keys" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787916 4622 cpu_manager.go:414] "containerMap: removing container" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787922 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787933 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787938 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b08100e-eb64-4016-983b-64acf8052259" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787945 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.787950 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788009 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="998d5375-3e23-44a4-83a3-b97150984c71" containerName="payload-processing" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788019 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788025 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde37f6-504c-4dd5-9764-65910dc63ca2" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788030 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a5f1eb-9331-425d-9e91-df7a100d4a3d" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788036 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" containerName="main" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788042 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" containerName="main" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788047 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788053 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d32316-94a5-4484-b67a-387c737461e3" containerName="istio-proxy" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788059 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="559092cc-213f-45b7-aeb6-6aa816959162" containerName="main" Oct 30 18:06:49.788133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.788068 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4f2cac-7bca-4af6-8137-721d7dff2276" containerName="revoke-keys" Oct 30 18:06:49.792857 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.792840 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.809040 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.809017 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:06:49.809246 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.809226 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:06:49.811962 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.811937 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh"] Oct 30 18:06:49.881605 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.881573 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.881810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.881661 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdvx\" (UniqueName: \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.881810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.881734 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.983132 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.983068 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdvx\" (UniqueName: \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.983132 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.983106 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.983569 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.983221 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.983894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.983874 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:49.985441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:49.985422 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:50.004453 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.004427 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdvx\" (UniqueName: \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\") pod \"maas-api-revoke-sub-2c2f19f3-7g2sh\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:50.106391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.106370 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:50.373767 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.373736 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3427dc4e-2da1-48d2-bfe3-859ab0d20550" path="/var/lib/kubelet/pods/3427dc4e-2da1-48d2-bfe3-859ab0d20550/volumes" Oct 30 18:06:50.374170 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.374156 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559092cc-213f-45b7-aeb6-6aa816959162" path="/var/lib/kubelet/pods/559092cc-213f-45b7-aeb6-6aa816959162/volumes" Oct 30 18:06:50.374611 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.374591 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c06b9a2-2375-4173-afd0-7049113c75e5" path="/var/lib/kubelet/pods/5c06b9a2-2375-4173-afd0-7049113c75e5/volumes" Oct 30 18:06:50.405773 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.405658 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh"] Oct 30 18:06:50.408207 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:06:50.408180 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb60840_9ff0_4fb1_ac0d_b4e0886868ec.slice/crio-25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6 WatchSource:0}: Error finding container 25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6: Status 404 returned error can't find the container with id 25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6 Oct 30 18:06:50.592687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:50.592651 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" event={"ID":"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec","Type":"ContainerStarted","Data":"25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6"} Oct 30 18:06:51.601456 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:51.601425 4622 generic.go:356] "Generic (PLEG): container finished" podID="0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" containerID="a210d825300779d56bcbb2a837d47aa6898af58675f499a6ea48135fd876420e" exitCode=0 Oct 30 18:06:51.601835 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:51.601463 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" event={"ID":"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec","Type":"ContainerDied","Data":"a210d825300779d56bcbb2a837d47aa6898af58675f499a6ea48135fd876420e"} Oct 30 18:06:52.609499 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.609472 4622 generic.go:356] "Generic (PLEG): container finished" podID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerID="2328794aea90130b4dc484873ffae9b702bae7eebde70f5d9310994907b74027" exitCode=0 Oct 30 18:06:52.609880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.609541 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerDied","Data":"2328794aea90130b4dc484873ffae9b702bae7eebde70f5d9310994907b74027"} Oct 30 18:06:52.609880 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.609603 4622 scope.go:122] "RemoveContainer" containerID="8726fba6b45aa836e3e86de4e407e2a44a7206d5a001517ac4c082a76d4f1044" Oct 30 18:06:52.761530 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.761467 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:52.785554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.785531 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:06:52.813194 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.813168 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\") pod \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " Oct 30 18:06:52.813410 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.813206 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\") pod \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " Oct 30 18:06:52.813410 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.813386 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\" (UniqueName: \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\") pod \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\" (UID: \"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec\") " Oct 30 18:06:52.813593 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.813570 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca" pod "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" (UID: "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:52.813746 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.813727 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.815197 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.815174 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca" pod "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" (UID: "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:06:52.815368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.815350 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx" pod "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" (UID: "0cb60840-9ff0-4fb1-ac0d-b4e0886868ec"). InnerVolumeSpecName "kube-api-access-4pdvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:52.914016 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.913984 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914180 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914073 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914180 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914115 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914190 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914288 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914238 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914383 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914318 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914383 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914350 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914502 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914412 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914502 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914423 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:52.914502 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914447 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\") pod \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\" (UID: \"70bb792b-b985-43cc-ac13-3ce3d7832dfb\") " Oct 30 18:06:52.914687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914496 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914843 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914854 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.914998 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915013 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdvx\" (UniqueName: \"kubernetes.io/projected/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-kube-api-access-4pdvx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915024 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915034 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915045 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915056 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:52.915436 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.915101 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:06:52.916370 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.916344 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:52.916477 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.916412 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:06:52.916582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.916566 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:52.916985 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:52.916971 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb" pod "70bb792b-b985-43cc-ac13-3ce3d7832dfb" (UID: "70bb792b-b985-43cc-ac13-3ce3d7832dfb"). InnerVolumeSpecName "kube-api-access-d5rwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:53.015788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.015720 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.015788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.015752 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.015788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.015763 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rwb\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-kube-api-access-d5rwb\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.015788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.015772 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.015788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.015780 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70bb792b-b985-43cc-ac13-3ce3d7832dfb-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:06:53.653819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.653759 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" event={"ID":"70bb792b-b985-43cc-ac13-3ce3d7832dfb","Type":"ContainerDied","Data":"3eca819b7b160b6016ec2a61e460eb89a1ee0bb050761ce5011d860076e4e729"} Oct 30 18:06:53.654479 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.653840 4622 scope.go:122] "RemoveContainer" containerID="2328794aea90130b4dc484873ffae9b702bae7eebde70f5d9310994907b74027" Oct 30 18:06:53.654479 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.653954 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm" Oct 30 18:06:53.660330 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.660297 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" event={"ID":"0cb60840-9ff0-4fb1-ac0d-b4e0886868ec","Type":"ContainerDied","Data":"25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6"} Oct 30 18:06:53.660591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.660558 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25498bf848ac42f62828e529b1f871da95b9e88c607a52eedfe810c9ce9e66e6" Oct 30 18:06:53.660874 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.660761 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh" Oct 30 18:06:53.715512 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.715481 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:06:53.725927 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:53.725763 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-584a12-openshift-default-77678bcfd9-8flrm"] Oct 30 18:06:54.374722 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:06:54.374685 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" path="/var/lib/kubelet/pods/70bb792b-b985-43cc-ac13-3ce3d7832dfb/volumes" Oct 30 18:06:54.533658 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:06:54.533627 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:07:17.286009 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:17.285969 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 18:07:17.287030 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:17.286238 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" containerID="cri-o://f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370" gracePeriod=30 Oct 30 18:07:22.563357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.563248 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 18:07:22.658366 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.658321 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.658566 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.658408 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.658566 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.658489 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.659441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659047 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:22.659441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659386 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.659752 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659723 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.659873 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659807 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.659873 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659852 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.660012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659922 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.660012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659968 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.660012 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.659996 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\") pod \"d16a824b-5141-48b3-aa25-a30297c44ffa\" (UID: \"d16a824b-5141-48b3-aa25-a30297c44ffa\") " Oct 30 18:07:22.660454 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.660434 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16a824b-5141-48b3-aa25-a30297c44ffa-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.660509 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.660460 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.661137 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.661086 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.661276 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.661116 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.661361 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.661309 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.668343 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.662772 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "kube-api-access-2k2th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:22.668343 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.663619 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:07:22.675564 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.675536 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:22.675737 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.675712 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token" pod "d16a824b-5141-48b3-aa25-a30297c44ffa" (UID: "d16a824b-5141-48b3-aa25-a30297c44ffa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:22.763578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763547 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763576 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763585 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763820 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763593 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16a824b-5141-48b3-aa25-a30297c44ffa-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763820 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763602 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k2th\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-kube-api-access-2k2th\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763820 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763611 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.763820 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.763622 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16a824b-5141-48b3-aa25-a30297c44ffa-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.843005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.842926 4622 generic.go:356] "Generic (PLEG): container finished" podID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerID="f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370" exitCode=0 Oct 30 18:07:22.843005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.842987 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" Oct 30 18:07:22.843005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.842998 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" event={"ID":"d16a824b-5141-48b3-aa25-a30297c44ffa","Type":"ContainerDied","Data":"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370"} Oct 30 18:07:22.843308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.843040 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f" event={"ID":"d16a824b-5141-48b3-aa25-a30297c44ffa","Type":"ContainerDied","Data":"3eb7412892235898ce700ea79f10972896b6acf29d410488853eee59f3ea465b"} Oct 30 18:07:22.843308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.843061 4622 scope.go:122] "RemoveContainer" containerID="f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370" Oct 30 18:07:22.853071 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.853051 4622 scope.go:122] "RemoveContainer" containerID="f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370" Oct 30 18:07:22.853377 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:22.853357 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370\": container with ID starting with f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370 not found: ID does not exist" containerID="f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370" Oct 30 18:07:22.853447 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.853387 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370"} err="failed to get container status \"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370\": rpc error: code = NotFound desc = could not find container \"f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370\": container with ID starting with f2a95fcd7801b752fe0a64d79469ff2096f2855b80dd967f10c64fc00afd5370 not found: ID does not exist" Oct 30 18:07:22.890109 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.890086 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 18:07:22.894014 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:22.893991 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-506124-openshift-default-6c7b97684f-knj6f"] Oct 30 18:07:24.343972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:24.343247 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:07:24.343972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:24.343568 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" containerID="cri-o://d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493" gracePeriod=30 Oct 30 18:07:24.373931 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:24.373899 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" path="/var/lib/kubelet/pods/d16a824b-5141-48b3-aa25-a30297c44ffa/volumes" Oct 30 18:07:24.541102 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:24.541069 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:07:29.602839 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.602811 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:07:29.732049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.731975 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732015 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732033 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732164 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732276 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732359 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732316 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732407 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732452 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732521 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:29.732551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732536 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\") pod \"685cc8ad-996f-4813-a663-0b9e875fe978\" (UID: \"685cc8ad-996f-4813-a663-0b9e875fe978\") " Oct 30 18:07:29.732766 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732670 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:29.732819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732768 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:29.732819 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732753 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:29.732993 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732966 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.732993 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.732993 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.733277 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.733006 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/685cc8ad-996f-4813-a663-0b9e875fe978-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.733277 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.733020 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.733277 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.733077 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:29.734597 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.734565 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "kube-api-access-fzsdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:29.735057 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.735027 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:29.735057 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.735043 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:07:29.735190 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.735116 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token" pod "685cc8ad-996f-4813-a663-0b9e875fe978" (UID: "685cc8ad-996f-4813-a663-0b9e875fe978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:29.834645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.834612 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/685cc8ad-996f-4813-a663-0b9e875fe978-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.834645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.834644 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.834840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.834655 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/685cc8ad-996f-4813-a663-0b9e875fe978-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.834840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.834679 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzsdm\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-kube-api-access-fzsdm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.834840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.834689 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/685cc8ad-996f-4813-a663-0b9e875fe978-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:29.891809 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.891779 4622 generic.go:356] "Generic (PLEG): container finished" podID="685cc8ad-996f-4813-a663-0b9e875fe978" containerID="d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493" exitCode=0 Oct 30 18:07:29.891985 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.891829 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" Oct 30 18:07:29.891985 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.891863 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerDied","Data":"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493"} Oct 30 18:07:29.891985 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.891950 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz" event={"ID":"685cc8ad-996f-4813-a663-0b9e875fe978","Type":"ContainerDied","Data":"1ca778d7a67f22590ba59e1eb027fe015ec4f6d5823e6f6666e7ceeba5555d49"} Oct 30 18:07:29.891985 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.891979 4622 scope.go:122] "RemoveContainer" containerID="d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493" Oct 30 18:07:29.902110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.902093 4622 scope.go:122] "RemoveContainer" containerID="2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2" Oct 30 18:07:29.910959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.910942 4622 scope.go:122] "RemoveContainer" containerID="d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493" Oct 30 18:07:29.911233 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:29.911214 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493\": container with ID starting with d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493 not found: ID does not exist" containerID="d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493" Oct 30 18:07:29.911323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.911241 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493"} err="failed to get container status \"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493\": rpc error: code = NotFound desc = could not find container \"d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493\": container with ID starting with d6a52c6da504c6ae134f3c89c2d941ab0ade4fd6345a9a8dde8d76a5c0c44493 not found: ID does not exist" Oct 30 18:07:29.911323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.911258 4622 scope.go:122] "RemoveContainer" containerID="2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2" Oct 30 18:07:29.911531 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:29.911511 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2\": container with ID starting with 2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2 not found: ID does not exist" containerID="2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2" Oct 30 18:07:29.911578 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.911540 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2"} err="failed to get container status \"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2\": rpc error: code = NotFound desc = could not find container \"2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2\": container with ID starting with 2ab3c8a1ddcc0b637910e6fb6586254a189a2ec430b5c61ee5933ffd38b58ab2 not found: ID does not exist" Oct 30 18:07:29.939394 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.939361 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:07:29.943340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:29.943321 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-96c514-openshift-default-7d9c746bdf-pmcmz"] Oct 30 18:07:30.374157 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:30.374119 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" path="/var/lib/kubelet/pods/685cc8ad-996f-4813-a663-0b9e875fe978/volumes" Oct 30 18:07:51.705206 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:51.705169 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 18:07:51.705922 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:51.705400 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" podUID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" containerName="maas-api" containerID="cri-o://49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b" gracePeriod=30 Oct 30 18:07:51.975221 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:51.975188 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 18:07:52.030729 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.030701 4622 generic.go:356] "Generic (PLEG): container finished" podID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" containerID="49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b" exitCode=0 Oct 30 18:07:52.030883 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.030751 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" Oct 30 18:07:52.030883 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.030757 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" event={"ID":"6eca3c38-3a21-4781-bbb2-03e8a27c9e88","Type":"ContainerDied","Data":"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b"} Oct 30 18:07:52.030997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.030893 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289" event={"ID":"6eca3c38-3a21-4781-bbb2-03e8a27c9e88","Type":"ContainerDied","Data":"bb2f6fc8c038dbe274bc3b4e0b898b728645e11a749f16a206df8c6df6d62082"} Oct 30 18:07:52.030997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.030937 4622 scope.go:122] "RemoveContainer" containerID="49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b" Oct 30 18:07:52.040217 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.040192 4622 scope.go:122] "RemoveContainer" containerID="49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b" Oct 30 18:07:52.040551 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:52.040526 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b\": container with ID starting with 49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b not found: ID does not exist" containerID="49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b" Oct 30 18:07:52.040670 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.040557 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b"} err="failed to get container status \"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b\": rpc error: code = NotFound desc = could not find container \"49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b\": container with ID starting with 49dbf2ff0025061b4eb3f59745eb99d1a7dc686cf6fe70a29d92a87f98925e8b not found: ID does not exist" Oct 30 18:07:52.055941 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.055921 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\") pod \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " Oct 30 18:07:52.056044 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.055991 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\") pod \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " Oct 30 18:07:52.056131 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.056117 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\") pod \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " Oct 30 18:07:52.056247 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.056225 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\" (UniqueName: \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\") pod \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\" (UID: \"6eca3c38-3a21-4781-bbb2-03e8a27c9e88\") " Oct 30 18:07:52.056384 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.056348 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca" pod "6eca3c38-3a21-4781-bbb2-03e8a27c9e88" (UID: "6eca3c38-3a21-4781-bbb2-03e8a27c9e88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:52.056623 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.056603 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:52.058032 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.058013 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs" pod "6eca3c38-3a21-4781-bbb2-03e8a27c9e88" (UID: "6eca3c38-3a21-4781-bbb2-03e8a27c9e88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:52.058166 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.058149 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls" pod "6eca3c38-3a21-4781-bbb2-03e8a27c9e88" (UID: "6eca3c38-3a21-4781-bbb2-03e8a27c9e88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:52.058212 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.058179 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k" pod "6eca3c38-3a21-4781-bbb2-03e8a27c9e88" (UID: "6eca3c38-3a21-4781-bbb2-03e8a27c9e88"). InnerVolumeSpecName "kube-api-access-dvm8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:52.157728 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.157695 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:52.157728 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.157727 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvm8k\" (UniqueName: \"kubernetes.io/projected/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-kube-api-access-dvm8k\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:52.157728 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.157736 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca3c38-3a21-4781-bbb2-03e8a27c9e88-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:07:52.374761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.374732 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 18:07:52.385013 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:52.384984 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-c31c73-6485b5585c-rq289"] Oct 30 18:07:54.373180 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:54.373143 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" path="/var/lib/kubelet/pods/6eca3c38-3a21-4781-bbb2-03e8a27c9e88/volumes" Oct 30 18:07:54.532445 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:07:54.532402 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:07:59.558177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:59.558136 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 18:07:59.558697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:07:59.558438 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" containerID="cri-o://12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2" gracePeriod=30 Oct 30 18:08:04.815175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.815147 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 18:08:04.974010 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.973938 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974010 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974001 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974020 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974051 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974102 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974134 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974174 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974221 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974255 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.974514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974313 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.974514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974328 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\") pod \"be4ca4c0-82e5-4322-96ee-81a62566f428\" (UID: \"be4ca4c0-82e5-4322-96ee-81a62566f428\") " Oct 30 18:08:04.974747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974597 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.974747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974712 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.974747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974732 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.974747 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974747 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:04.974917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974690 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:04.974917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.974781 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.976411 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.976389 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:04.976539 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.976520 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "kube-api-access-f84tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:04.976802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.976779 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:04.976902 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:04.976798 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo" pod "be4ca4c0-82e5-4322-96ee-81a62566f428" (UID: "be4ca4c0-82e5-4322-96ee-81a62566f428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:08:05.075591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075560 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84tz\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-kube-api-access-f84tz\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.075591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075588 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.075761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075599 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.075761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075608 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.075761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075617 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4ca4c0-82e5-4322-96ee-81a62566f428-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.075761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.075625 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4ca4c0-82e5-4322-96ee-81a62566f428-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:05.107785 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.107759 4622 generic.go:356] "Generic (PLEG): container finished" podID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerID="12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2" exitCode=0 Oct 30 18:08:05.107920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.107811 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" Oct 30 18:08:05.107920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.107848 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" event={"ID":"be4ca4c0-82e5-4322-96ee-81a62566f428","Type":"ContainerDied","Data":"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2"} Oct 30 18:08:05.107920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.107903 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr" event={"ID":"be4ca4c0-82e5-4322-96ee-81a62566f428","Type":"ContainerDied","Data":"20cd508da595c6bd39ff2f2d5b07f70241dc6931587c991866f35fc47bd3b6e2"} Oct 30 18:08:05.108042 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.107933 4622 scope.go:122] "RemoveContainer" containerID="12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2" Oct 30 18:08:05.118217 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.118203 4622 scope.go:122] "RemoveContainer" containerID="12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2" Oct 30 18:08:05.118554 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:08:05.118536 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2\": container with ID starting with 12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2 not found: ID does not exist" containerID="12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2" Oct 30 18:08:05.118601 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.118559 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2"} err="failed to get container status \"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2\": rpc error: code = NotFound desc = could not find container \"12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2\": container with ID starting with 12a8200e388c29edbe3426a5c8f75a6d139588576fd48cd142cbbec0c80564d2 not found: ID does not exist" Oct 30 18:08:05.157949 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.157926 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 18:08:05.168302 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:05.168248 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-c31c73-openshift-default-65b88bfd67-5k5rr"] Oct 30 18:08:06.373358 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:06.373326 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" path="/var/lib/kubelet/pods/be4ca4c0-82e5-4322-96ee-81a62566f428/volumes" Oct 30 18:08:13.013473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.013433 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 18:08:13.013942 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.013705 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" podUID="67a256b9-aef1-4064-b94e-36957242d793" containerName="maas-api" containerID="cri-o://22404844d3863ebca50f1a1b51c7217cf6599ea0770cd9d3db83e579cb9b9a21" gracePeriod=30 Oct 30 18:08:13.161850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.161818 4622 generic.go:356] "Generic (PLEG): container finished" podID="67a256b9-aef1-4064-b94e-36957242d793" containerID="22404844d3863ebca50f1a1b51c7217cf6599ea0770cd9d3db83e579cb9b9a21" exitCode=0 Oct 30 18:08:13.162014 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.161905 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" event={"ID":"67a256b9-aef1-4064-b94e-36957242d793","Type":"ContainerDied","Data":"22404844d3863ebca50f1a1b51c7217cf6599ea0770cd9d3db83e579cb9b9a21"} Oct 30 18:08:13.285551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.285520 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 18:08:13.449434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.449392 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\") pod \"67a256b9-aef1-4064-b94e-36957242d793\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " Oct 30 18:08:13.449632 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.449614 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\") pod \"67a256b9-aef1-4064-b94e-36957242d793\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " Oct 30 18:08:13.449733 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.449716 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") pod \"67a256b9-aef1-4064-b94e-36957242d793\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " Oct 30 18:08:13.449824 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.449754 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\" (UniqueName: \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\") pod \"67a256b9-aef1-4064-b94e-36957242d793\" (UID: \"67a256b9-aef1-4064-b94e-36957242d793\") " Oct 30 18:08:13.450002 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.449977 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca" pod "67a256b9-aef1-4064-b94e-36957242d793" (UID: "67a256b9-aef1-4064-b94e-36957242d793"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:13.450145 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.450129 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a256b9-aef1-4064-b94e-36957242d793-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:13.451886 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.451858 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx" pod "67a256b9-aef1-4064-b94e-36957242d793" (UID: "67a256b9-aef1-4064-b94e-36957242d793"). InnerVolumeSpecName "kube-api-access-57cvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:13.452099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.452079 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs" pod "67a256b9-aef1-4064-b94e-36957242d793" (UID: "67a256b9-aef1-4064-b94e-36957242d793"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:13.452171 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.452124 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls" pod "67a256b9-aef1-4064-b94e-36957242d793" (UID: "67a256b9-aef1-4064-b94e-36957242d793"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:13.551536 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.551509 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57cvx\" (UniqueName: \"kubernetes.io/projected/67a256b9-aef1-4064-b94e-36957242d793-kube-api-access-57cvx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:13.551536 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.551535 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:13.551692 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:13.551544 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a256b9-aef1-4064-b94e-36957242d793-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:14.168715 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.168685 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" Oct 30 18:08:14.169120 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.168719 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc" event={"ID":"67a256b9-aef1-4064-b94e-36957242d793","Type":"ContainerDied","Data":"dea00a83ed1df896461d627e0a7b7e629b79a4caa0ecad857aeca47ac27c6e9e"} Oct 30 18:08:14.169120 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.168783 4622 scope.go:122] "RemoveContainer" containerID="22404844d3863ebca50f1a1b51c7217cf6599ea0770cd9d3db83e579cb9b9a21" Oct 30 18:08:14.214830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.214800 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 18:08:14.225226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.225202 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d53ec1-79f576fbf9-qnhmc"] Oct 30 18:08:14.373083 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:14.373054 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a256b9-aef1-4064-b94e-36957242d793" path="/var/lib/kubelet/pods/67a256b9-aef1-4064-b94e-36957242d793/volumes" Oct 30 18:08:24.532601 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:08:24.532561 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:08:26.172253 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:26.172213 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 18:08:26.179832 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:26.179803 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 18:08:39.685440 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685403 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2"] Oct 30 18:08:39.685832 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685807 4622 cpu_manager.go:414] "containerMap: removing container" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:08:39.685832 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685818 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685839 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685844 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685854 4622 cpu_manager.go:414] "containerMap: removing container" podUID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" containerName="maas-api" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685859 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" containerName="maas-api" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685867 4622 cpu_manager.go:414] "containerMap: removing container" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685872 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685878 4622 cpu_manager.go:414] "containerMap: removing container" podUID="67a256b9-aef1-4064-b94e-36957242d793" containerName="maas-api" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685882 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a256b9-aef1-4064-b94e-36957242d793" containerName="maas-api" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685889 4622 cpu_manager.go:414] "containerMap: removing container" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685895 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685901 4622 cpu_manager.go:414] "containerMap: removing container" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685906 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.685910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685914 4622 cpu_manager.go:414] "containerMap: removing container" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685919 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56c400d2-e231-4269-bced-1a5235ae7cf2" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685935 4622 cpu_manager.go:414] "containerMap: removing container" podUID="0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" containerName="revoke-keys" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685940 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" containerName="revoke-keys" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.685994 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16a824b-5141-48b3-aa25-a30297c44ffa" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686001 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eca3c38-3a21-4781-bbb2-03e8a27c9e88" containerName="maas-api" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686009 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4ca4c0-82e5-4322-96ee-81a62566f428" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686013 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a256b9-aef1-4064-b94e-36957242d793" containerName="maas-api" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686017 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686022 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="685cc8ad-996f-4813-a663-0b9e875fe978" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686030 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:08:39.686347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.686035 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" containerName="revoke-keys" Oct 30 18:08:39.690610 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.690588 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.709996 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.709963 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2"] Oct 30 18:08:39.763508 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.763481 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.763677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.763529 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drg8x\" (UniqueName: \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.763746 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.763710 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.796660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.796630 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:08:39.796660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.796640 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:08:39.864400 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.864376 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.864511 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.864422 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.864511 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.864442 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drg8x\" (UniqueName: \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.865010 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.864989 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.866679 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.866657 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:39.885532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:39.885509 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drg8x\" (UniqueName: \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\") pod \"maas-api-revoke-sub-36ef1e68-z22x2\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:40.007311 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:40.007223 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:40.308211 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:40.308182 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2"] Oct 30 18:08:40.310198 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:08:40.310166 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbe6c86_7fa1_45f9_aa21_da4e59583f74.slice/crio-7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0 WatchSource:0}: Error finding container 7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0: Status 404 returned error can't find the container with id 7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0 Oct 30 18:08:41.193373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.193306 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" containerID="cri-o://62a611d1503b212ee7798b9674a70e7d6735a373028f4a5ff16ab4cc245ce7da" gracePeriod=45 Oct 30 18:08:41.205686 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.205656 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" containerID="cri-o://dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04" gracePeriod=45 Oct 30 18:08:41.318699 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.318667 4622 generic.go:356] "Generic (PLEG): container finished" podID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerID="62a611d1503b212ee7798b9674a70e7d6735a373028f4a5ff16ab4cc245ce7da" exitCode=0 Oct 30 18:08:41.318806 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.318710 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" event={"ID":"fbf0b8f4-42ed-4167-9777-ec1f917d16b6","Type":"ContainerDied","Data":"62a611d1503b212ee7798b9674a70e7d6735a373028f4a5ff16ab4cc245ce7da"} Oct 30 18:08:41.320560 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.320534 4622 generic.go:356] "Generic (PLEG): container finished" podID="ddbe6c86-7fa1-45f9-aa21-da4e59583f74" containerID="b29676dfd48a2a40106be21b750b8847716426489b34bd076c28b5f49b5fea56" exitCode=0 Oct 30 18:08:41.320676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.320615 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" event={"ID":"ddbe6c86-7fa1-45f9-aa21-da4e59583f74","Type":"ContainerDied","Data":"b29676dfd48a2a40106be21b750b8847716426489b34bd076c28b5f49b5fea56"} Oct 30 18:08:41.320676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.320660 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" event={"ID":"ddbe6c86-7fa1-45f9-aa21-da4e59583f74","Type":"ContainerStarted","Data":"7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0"} Oct 30 18:08:41.469504 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.469478 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 18:08:41.472425 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.472404 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 18:08:41.581929 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.581895 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.581951 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.581985 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\" (UniqueName: \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582014 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.582110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582056 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.582110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582107 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582141 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\" (UniqueName: \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582179 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582210 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582231 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582349 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582409 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582469 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582505 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582537 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582613 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\") pod \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\" (UID: \"fbf0b8f4-42ed-4167-9777-ec1f917d16b6\") " Oct 30 18:08:41.582845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582657 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\" (UniqueName: \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\") pod \"d294de07-d345-411b-ae06-c23263f5c261\" (UID: \"d294de07-d345-411b-ae06-c23263f5c261\") " Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.583161 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.583182 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.583197 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.583209 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.583220 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.583500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.582745 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.585020 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.584987 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.585020 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.585002 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:41.585153 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.584989 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:41.585223 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.585209 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:41.585355 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.585341 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw" pod "fbf0b8f4-42ed-4167-9777-ec1f917d16b6" (UID: "fbf0b8f4-42ed-4167-9777-ec1f917d16b6"). InnerVolumeSpecName "kube-api-access-698zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:41.585691 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.585674 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8" pod "d294de07-d345-411b-ae06-c23263f5c261" (UID: "d294de07-d345-411b-ae06-c23263f5c261"). InnerVolumeSpecName "kube-api-access-bpzx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:41.684028 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.683998 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684028 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684024 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d294de07-d345-411b-ae06-c23263f5c261-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684028 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684034 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684228 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684044 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-698zw\" (UniqueName: \"kubernetes.io/projected/fbf0b8f4-42ed-4167-9777-ec1f917d16b6-kube-api-access-698zw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684228 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684055 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzx8\" (UniqueName: \"kubernetes.io/projected/d294de07-d345-411b-ae06-c23263f5c261-kube-api-access-bpzx8\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684228 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684063 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:41.684228 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:41.684072 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d294de07-d345-411b-ae06-c23263f5c261-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:42.326896 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.326863 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" Oct 30 18:08:42.327422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.326902 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf" event={"ID":"fbf0b8f4-42ed-4167-9777-ec1f917d16b6","Type":"ContainerDied","Data":"5278132e8ffedb9a7ee2fb771fc0dfd86a460ed97ba4bd02a0926f3242d21949"} Oct 30 18:08:42.327422 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.326954 4622 scope.go:122] "RemoveContainer" containerID="62a611d1503b212ee7798b9674a70e7d6735a373028f4a5ff16ab4cc245ce7da" Oct 30 18:08:42.328402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.328376 4622 generic.go:356] "Generic (PLEG): container finished" podID="d294de07-d345-411b-ae06-c23263f5c261" containerID="dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04" exitCode=0 Oct 30 18:08:42.328547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.328404 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" event={"ID":"d294de07-d345-411b-ae06-c23263f5c261","Type":"ContainerDied","Data":"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04"} Oct 30 18:08:42.328547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.328448 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" event={"ID":"d294de07-d345-411b-ae06-c23263f5c261","Type":"ContainerDied","Data":"fa5c04920bdb8a90dfdfcd7ed0497a0d238fc4533a2d0dc4f2d5070e8f0b1613"} Oct 30 18:08:42.328547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.328418 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj" Oct 30 18:08:42.336561 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.336541 4622 scope.go:122] "RemoveContainer" containerID="dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04" Oct 30 18:08:42.345469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.345450 4622 scope.go:122] "RemoveContainer" containerID="dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04" Oct 30 18:08:42.345753 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:08:42.345732 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04\": container with ID starting with dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04 not found: ID does not exist" containerID="dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04" Oct 30 18:08:42.345841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.345765 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04"} err="failed to get container status \"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04\": rpc error: code = NotFound desc = could not find container \"dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04\": container with ID starting with dd8717f2fb0ad1f91372cddca3308bee428b62b6dd1076d780f84408e79e2b04 not found: ID does not exist" Oct 30 18:08:42.393317 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.393288 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 18:08:42.405325 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.405296 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/simulator-w0-27567a-kserve-5747f855d7-5twnj"] Oct 30 18:08:42.427735 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.427699 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 18:08:42.433367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.433342 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-27567a/premium-w0-27567a-kserve-85f886465d-c7xtf"] Oct 30 18:08:42.456948 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.456928 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:42.491049 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.491028 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\" (UniqueName: \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\") pod \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " Oct 30 18:08:42.491178 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.491148 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\") pod \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " Oct 30 18:08:42.491244 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.491197 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\") pod \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\" (UID: \"ddbe6c86-7fa1-45f9-aa21-da4e59583f74\") " Oct 30 18:08:42.491647 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.491619 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca" pod "ddbe6c86-7fa1-45f9-aa21-da4e59583f74" (UID: "ddbe6c86-7fa1-45f9-aa21-da4e59583f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:42.493110 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.493081 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x" pod "ddbe6c86-7fa1-45f9-aa21-da4e59583f74" (UID: "ddbe6c86-7fa1-45f9-aa21-da4e59583f74"). InnerVolumeSpecName "kube-api-access-drg8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:42.493237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.493213 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca" pod "ddbe6c86-7fa1-45f9-aa21-da4e59583f74" (UID: "ddbe6c86-7fa1-45f9-aa21-da4e59583f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:42.592573 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.592515 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:42.592573 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.592540 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drg8x\" (UniqueName: \"kubernetes.io/projected/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-kube-api-access-drg8x\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:42.592573 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:42.592552 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddbe6c86-7fa1-45f9-aa21-da4e59583f74-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:43.336551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:43.336524 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" Oct 30 18:08:43.336551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:43.336531 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2" event={"ID":"ddbe6c86-7fa1-45f9-aa21-da4e59583f74","Type":"ContainerDied","Data":"7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0"} Oct 30 18:08:43.337091 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:43.336589 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7caa65e6ec042544521f8f8d143f05fb119b6d3eb9c377127507a8ce77afe0" Oct 30 18:08:44.373535 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:44.373502 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d294de07-d345-411b-ae06-c23263f5c261" path="/var/lib/kubelet/pods/d294de07-d345-411b-ae06-c23263f5c261/volumes" Oct 30 18:08:44.373911 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:44.373901 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" path="/var/lib/kubelet/pods/fbf0b8f4-42ed-4167-9777-ec1f917d16b6/volumes" Oct 30 18:08:54.533645 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:08:54.533559 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:08:55.193712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.193680 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:08:55.193939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.193910 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" podUID="f792563d-e2b2-418a-9eb7-b12dd474bf51" containerName="maas-api" containerID="cri-o://ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a" gracePeriod=30 Oct 30 18:08:55.648069 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.648040 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:08:55.734477 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.734395 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\" (UniqueName: \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\") pod \"f792563d-e2b2-418a-9eb7-b12dd474bf51\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " Oct 30 18:08:55.734477 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.734448 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\") pod \"f792563d-e2b2-418a-9eb7-b12dd474bf51\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " Oct 30 18:08:55.734701 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.734569 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") pod \"f792563d-e2b2-418a-9eb7-b12dd474bf51\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " Oct 30 18:08:55.734701 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.734621 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\") pod \"f792563d-e2b2-418a-9eb7-b12dd474bf51\" (UID: \"f792563d-e2b2-418a-9eb7-b12dd474bf51\") " Oct 30 18:08:55.735324 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.735292 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca" pod "f792563d-e2b2-418a-9eb7-b12dd474bf51" (UID: "f792563d-e2b2-418a-9eb7-b12dd474bf51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:55.736923 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.736899 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs" pod "f792563d-e2b2-418a-9eb7-b12dd474bf51" (UID: "f792563d-e2b2-418a-9eb7-b12dd474bf51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:55.737144 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.737117 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv" pod "f792563d-e2b2-418a-9eb7-b12dd474bf51" (UID: "f792563d-e2b2-418a-9eb7-b12dd474bf51"). InnerVolumeSpecName "kube-api-access-bfbgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:55.737195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.737169 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls" pod "f792563d-e2b2-418a-9eb7-b12dd474bf51" (UID: "f792563d-e2b2-418a-9eb7-b12dd474bf51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:55.835958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.835921 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfbgv\" (UniqueName: \"kubernetes.io/projected/f792563d-e2b2-418a-9eb7-b12dd474bf51-kube-api-access-bfbgv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:55.835958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.835954 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:55.835958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.835963 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f792563d-e2b2-418a-9eb7-b12dd474bf51-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:55.836251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:55.835971 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f792563d-e2b2-418a-9eb7-b12dd474bf51-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:08:56.414441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.414410 4622 generic.go:356] "Generic (PLEG): container finished" podID="f792563d-e2b2-418a-9eb7-b12dd474bf51" containerID="ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a" exitCode=0 Oct 30 18:08:56.414599 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.414454 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" Oct 30 18:08:56.414599 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.414498 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" event={"ID":"f792563d-e2b2-418a-9eb7-b12dd474bf51","Type":"ContainerDied","Data":"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a"} Oct 30 18:08:56.414599 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.414552 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd" event={"ID":"f792563d-e2b2-418a-9eb7-b12dd474bf51","Type":"ContainerDied","Data":"d335b474a74c355a134c6c72d8939bf326e613bd5383a12e4d030eab64de7889"} Oct 30 18:08:56.414599 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.414583 4622 scope.go:122] "RemoveContainer" containerID="ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a" Oct 30 18:08:56.424213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.424186 4622 scope.go:122] "RemoveContainer" containerID="ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a" Oct 30 18:08:56.424548 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:08:56.424519 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a\": container with ID starting with ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a not found: ID does not exist" containerID="ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a" Oct 30 18:08:56.424651 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.424563 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a"} err="failed to get container status \"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a\": rpc error: code = NotFound desc = could not find container \"ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a\": container with ID starting with ba45b4749046d789c66a8d2fecb0c1701d88e63b7ac6cdf8d9fdcb76e2cd0f8a not found: ID does not exist" Oct 30 18:08:56.462894 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.462867 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:08:56.472907 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:56.472880 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95f55d-75b76dd787-8vxkd"] Oct 30 18:08:58.372475 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:08:58.372430 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f792563d-e2b2-418a-9eb7-b12dd474bf51" path="/var/lib/kubelet/pods/f792563d-e2b2-418a-9eb7-b12dd474bf51/volumes" Oct 30 18:09:24.533393 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:09:24.533354 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:09:45.733237 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733202 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq"] Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733594 4622 cpu_manager.go:414] "containerMap: removing container" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733606 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733619 4622 cpu_manager.go:414] "containerMap: removing container" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733624 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733637 4622 cpu_manager.go:414] "containerMap: removing container" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733642 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733665 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f792563d-e2b2-418a-9eb7-b12dd474bf51" containerName="maas-api" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733669 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f792563d-e2b2-418a-9eb7-b12dd474bf51" containerName="maas-api" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733680 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ddbe6c86-7fa1-45f9-aa21-da4e59583f74" containerName="revoke-keys" Oct 30 18:09:45.733716 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733685 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddbe6c86-7fa1-45f9-aa21-da4e59583f74" containerName="revoke-keys" Oct 30 18:09:45.734034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733736 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf0b8f4-42ed-4167-9777-ec1f917d16b6" containerName="main" Oct 30 18:09:45.734034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733749 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f792563d-e2b2-418a-9eb7-b12dd474bf51" containerName="maas-api" Oct 30 18:09:45.734034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733756 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddbe6c86-7fa1-45f9-aa21-da4e59583f74" containerName="revoke-keys" Oct 30 18:09:45.734034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733762 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bb792b-b985-43cc-ac13-3ce3d7832dfb" containerName="istio-proxy" Oct 30 18:09:45.734034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.733768 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="d294de07-d345-411b-ae06-c23263f5c261" containerName="main" Oct 30 18:09:45.736674 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.736656 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.756026 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.756001 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq"] Oct 30 18:09:45.830541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.830513 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:09:45.830692 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.830629 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:09:45.875947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.875920 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4v7\" (UniqueName: \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.876061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.875967 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.876125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.876088 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.976760 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.976732 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4v7\" (UniqueName: \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.976891 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.976776 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.976891 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.976841 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.977476 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.977457 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.979244 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.979220 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:45.997929 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:45.997868 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4v7\" (UniqueName: \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\") pod \"maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:46.048949 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:46.048928 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:46.342536 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:46.342507 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq"] Oct 30 18:09:46.345412 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:09:46.345379 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7e2fd4_3c4c_4cb8_99b0_fe6c8a595d35.slice/crio-0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e WatchSource:0}: Error finding container 0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e: Status 404 returned error can't find the container with id 0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e Oct 30 18:09:46.693934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:46.693905 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" event={"ID":"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35","Type":"ContainerStarted","Data":"c25dc4322c4c9da216bed8c38994d09f8325c96af2056f5496303f3f8fb94fe9"} Oct 30 18:09:46.694054 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:46.693952 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" event={"ID":"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35","Type":"ContainerStarted","Data":"0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e"} Oct 30 18:09:46.730500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:46.730454 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" podStartSLOduration=1.730437524 podStartE2EDuration="1.730437524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:46.730374973 +0000 UTC m=+3414.507846928" watchObservedRunningTime="2026-10-30 18:09:46.730437524 +0000 UTC m=+3414.507909478" Oct 30 18:09:47.701781 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:47.701744 4622 generic.go:356] "Generic (PLEG): container finished" podID="8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" containerID="c25dc4322c4c9da216bed8c38994d09f8325c96af2056f5496303f3f8fb94fe9" exitCode=0 Oct 30 18:09:47.702193 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:47.701806 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" event={"ID":"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35","Type":"ContainerDied","Data":"c25dc4322c4c9da216bed8c38994d09f8325c96af2056f5496303f3f8fb94fe9"} Oct 30 18:09:48.839853 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:48.839827 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:49.004815 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.004743 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\") pod \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " Oct 30 18:09:49.004959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.004899 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\" (UniqueName: \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\") pod \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " Oct 30 18:09:49.004959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.004941 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\") pod \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\" (UID: \"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35\") " Oct 30 18:09:49.005314 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.005288 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca" pod "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" (UID: "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:09:49.006806 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.006786 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca" pod "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" (UID: "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:49.006947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.006927 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7" pod "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" (UID: "8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35"). InnerVolumeSpecName "kube-api-access-nc4v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:49.106225 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.106200 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4v7\" (UniqueName: \"kubernetes.io/projected/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-kube-api-access-nc4v7\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:09:49.106225 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.106222 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:09:49.106383 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.106232 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:09:49.713541 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.713514 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" Oct 30 18:09:49.713688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.713549 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq" event={"ID":"8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35","Type":"ContainerDied","Data":"0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e"} Oct 30 18:09:49.713688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:49.713593 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0afa94a6d926e3134c342d2b52d430fdca6904cbcef12e07bd19acdebd23632e" Oct 30 18:09:54.534723 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:09:54.534682 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:09:56.664792 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:56.664760 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq"] Oct 30 18:09:56.676077 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:56.676052 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d323f5-fc9fq"] Oct 30 18:09:58.372881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:09:58.372848 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" path="/var/lib/kubelet/pods/8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35/volumes" Oct 30 18:10:24.536498 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:10:24.536422 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:10:54.535652 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:10:54.535614 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:10:55.603843 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:10:55.603804 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 18:10:55.604525 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:10:55.604063 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" containerID="cri-o://5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3" gracePeriod=30 Oct 30 18:11:00.859911 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.859888 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 18:11:00.951605 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951539 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951605 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951598 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951634 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951655 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951682 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951716 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.951799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951754 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.952080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951799 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.952080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.951828 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\") pod \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\" (UID: \"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8\") " Oct 30 18:11:00.952080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952053 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:00.952293 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952161 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:00.952293 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952178 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:00.952293 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952228 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:00.952582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952296 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:00.952582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.952463 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:00.953984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.953960 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:00.954251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.954224 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:00.954371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.954299 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:11:00.954416 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:00.954373 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv" pod "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" (UID: "c31ffeb5-1de8-4da9-9140-5ed5cd104eb8"). InnerVolumeSpecName "kube-api-access-2tgzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:01.053647 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053613 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053647 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053636 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053647 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053649 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053658 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053673 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053683 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053692 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.053918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.053700 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tgzv\" (UniqueName: \"kubernetes.io/projected/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8-kube-api-access-2tgzv\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:01.071174 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.071151 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 18:11:01.085379 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.085352 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 18:11:01.122982 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.122948 4622 generic.go:356] "Generic (PLEG): container finished" podID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerID="5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3" exitCode=0 Oct 30 18:11:01.123133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.123000 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" event={"ID":"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8","Type":"ContainerDied","Data":"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3"} Oct 30 18:11:01.123133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.123003 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" Oct 30 18:11:01.123133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.123077 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz" event={"ID":"c31ffeb5-1de8-4da9-9140-5ed5cd104eb8","Type":"ContainerDied","Data":"3eb16950b3a26fb536c3996b1159a14205be1eaf24a906bca7ce7fb1148d0941"} Oct 30 18:11:01.123133 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.123113 4622 scope.go:122] "RemoveContainer" containerID="5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3" Oct 30 18:11:01.133670 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.133650 4622 scope.go:122] "RemoveContainer" containerID="5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3" Oct 30 18:11:01.133969 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:01.133947 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3\": container with ID starting with 5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3 not found: ID does not exist" containerID="5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3" Oct 30 18:11:01.134042 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.133981 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3"} err="failed to get container status \"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3\": rpc error: code = NotFound desc = could not find container \"5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3\": container with ID starting with 5dc10dbe6da49ebccc1aeaf5a774c136881513a04e7102ef000a120c9aa22be3 not found: ID does not exist" Oct 30 18:11:01.190751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.190726 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 18:11:01.195042 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:01.195020 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a23c98-openshift-default-7f9c8f49cc-mhvxz"] Oct 30 18:11:02.372756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:02.372720 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" path="/var/lib/kubelet/pods/c31ffeb5-1de8-4da9-9140-5ed5cd104eb8/volumes" Oct 30 18:11:06.746664 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:06.746630 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 18:11:06.762934 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:06.762901 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 18:11:11.747143 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:11.747088 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" podUID="564b66d8-b77c-4fb2-b34a-69e15936452e" containerName="payload-processing" containerID="cri-o://d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18" gracePeriod=25 Oct 30 18:11:11.763628 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:11.763600 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" containerID="cri-o://80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0" gracePeriod=25 Oct 30 18:11:11.988385 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:11.988356 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 18:11:12.046443 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.046419 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\") pod \"564b66d8-b77c-4fb2-b34a-69e15936452e\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " Oct 30 18:11:12.046584 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.046516 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\" (UniqueName: \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\") pod \"564b66d8-b77c-4fb2-b34a-69e15936452e\" (UID: \"564b66d8-b77c-4fb2-b34a-69e15936452e\") " Oct 30 18:11:12.046797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.046773 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume" pod "564b66d8-b77c-4fb2-b34a-69e15936452e" (UID: "564b66d8-b77c-4fb2-b34a-69e15936452e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:12.048603 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.048572 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm" pod "564b66d8-b77c-4fb2-b34a-69e15936452e" (UID: "564b66d8-b77c-4fb2-b34a-69e15936452e"). InnerVolumeSpecName "kube-api-access-lpxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:12.147494 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.147471 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxwm\" (UniqueName: \"kubernetes.io/projected/564b66d8-b77c-4fb2-b34a-69e15936452e-kube-api-access-lpxwm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.147494 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.147492 4622 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564b66d8-b77c-4fb2-b34a-69e15936452e-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:12.188965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.188938 4622 generic.go:356] "Generic (PLEG): container finished" podID="564b66d8-b77c-4fb2-b34a-69e15936452e" containerID="d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18" exitCode=0 Oct 30 18:11:12.189059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.188965 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" event={"ID":"564b66d8-b77c-4fb2-b34a-69e15936452e","Type":"ContainerDied","Data":"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18"} Oct 30 18:11:12.189059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.188984 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" Oct 30 18:11:12.189059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.189019 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms" event={"ID":"564b66d8-b77c-4fb2-b34a-69e15936452e","Type":"ContainerDied","Data":"e2d0f3b72882be01044eeb77675481abef02c77c62ec3ae3cc1a5f5e1448e965"} Oct 30 18:11:12.189059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.189042 4622 scope.go:122] "RemoveContainer" containerID="d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18" Oct 30 18:11:12.198366 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.198239 4622 scope.go:122] "RemoveContainer" containerID="d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18" Oct 30 18:11:12.198559 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:12.198541 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18\": container with ID starting with d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18 not found: ID does not exist" containerID="d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18" Oct 30 18:11:12.198620 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.198567 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18"} err="failed to get container status \"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18\": rpc error: code = NotFound desc = could not find container \"d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18\": container with ID starting with d5b77799f554b2d72017f65ff2c6f67d508b5c06161e0ab212452eb2499cfd18 not found: ID does not exist" Oct 30 18:11:12.234458 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.234437 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 18:11:12.238812 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.238787 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0e7403-774467cdcb-6jwms"] Oct 30 18:11:12.373970 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:12.373910 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564b66d8-b77c-4fb2-b34a-69e15936452e" path="/var/lib/kubelet/pods/564b66d8-b77c-4fb2-b34a-69e15936452e/volumes" Oct 30 18:11:13.162502 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.162469 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 18:11:13.162904 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.162649 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerName="discovery" containerID="cri-o://f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc" gracePeriod=30 Oct 30 18:11:13.412756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.412693 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 18:11:13.458095 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.458059 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:13.558831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.558806 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.558968 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.558896 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.558968 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.558946 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.559054 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.558978 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.559054 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.559030 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.559154 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.559058 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.559154 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.559112 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\") pod \"b4cf8876-b173-4519-8887-e9acf5c4005b\" (UID: \"b4cf8876-b173-4519-8887-e9acf5c4005b\") " Oct 30 18:11:13.559808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.559763 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:13.561411 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561383 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:13.561529 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561415 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "kube-api-access-hxtkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:13.561679 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561651 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:13.561797 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561725 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:13.561885 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561865 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:13.561948 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.561891 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert" pod "b4cf8876-b173-4519-8887-e9acf5c4005b" (UID: "b4cf8876-b173-4519-8887-e9acf5c4005b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:13.660495 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660469 4622 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-ca-configmap\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660495 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660492 4622 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-kubeconfig\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660503 4622 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-cacerts\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660512 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxtkt\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-kube-api-access-hxtkt\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660526 4622 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cf8876-b173-4519-8887-e9acf5c4005b-local-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660534 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:13.660622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:13.660543 4622 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4cf8876-b173-4519-8887-e9acf5c4005b-istio-csr-dns-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:14.202105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.202070 4622 generic.go:356] "Generic (PLEG): container finished" podID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerID="f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc" exitCode=0 Oct 30 18:11:14.202105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.202106 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" event={"ID":"b4cf8876-b173-4519-8887-e9acf5c4005b","Type":"ContainerDied","Data":"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc"} Oct 30 18:11:14.202707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.202121 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" Oct 30 18:11:14.202707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.202141 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h" event={"ID":"b4cf8876-b173-4519-8887-e9acf5c4005b","Type":"ContainerDied","Data":"104c99cce5f8d01a61e8a63ce41ac2bfbe50df73dca71657cc9016cedcd390d2"} Oct 30 18:11:14.202707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.202160 4622 scope.go:122] "RemoveContainer" containerID="f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc" Oct 30 18:11:14.215009 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.214992 4622 scope.go:122] "RemoveContainer" containerID="f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc" Oct 30 18:11:14.215284 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:14.215244 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc\": container with ID starting with f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc not found: ID does not exist" containerID="f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc" Oct 30 18:11:14.215338 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.215300 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc"} err="failed to get container status \"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc\": rpc error: code = NotFound desc = could not find container \"f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc\": container with ID starting with f8f5be2778c5393a7789cbe8df214c37dbee422e9f3713527998ae9e78ded9dc not found: ID does not exist" Oct 30 18:11:14.266752 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.266726 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 18:11:14.277388 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.277363 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kz8h"] Oct 30 18:11:14.372186 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:14.372162 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" path="/var/lib/kubelet/pods/b4cf8876-b173-4519-8887-e9acf5c4005b/volumes" Oct 30 18:11:16.090646 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.090606 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" containerID="cri-o://782f385df7c343f199284c11237e5215e3aa882c883cb40b23494c633c8ee4b6" gracePeriod=45 Oct 30 18:11:16.104727 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.104671 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" containerID="cri-o://83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8" gracePeriod=45 Oct 30 18:11:16.216200 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.216167 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": dial tcp 10.134.0.77:8000: connect: connection refused" Oct 30 18:11:16.216736 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.216705 4622 generic.go:356] "Generic (PLEG): container finished" podID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerID="782f385df7c343f199284c11237e5215e3aa882c883cb40b23494c633c8ee4b6" exitCode=0 Oct 30 18:11:16.216837 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.216741 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" event={"ID":"2cb8de23-92b8-4bc0-8c35-16e51709a6a2","Type":"ContainerDied","Data":"782f385df7c343f199284c11237e5215e3aa882c883cb40b23494c633c8ee4b6"} Oct 30 18:11:16.349397 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.349339 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 18:11:16.369841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.369820 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 18:11:16.484574 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484543 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\" (UniqueName: \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.484574 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484577 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.484808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484607 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.484808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484645 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.484808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484722 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.484808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484754 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.484808 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484794 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484828 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484854 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484867 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484885 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\" (UniqueName: \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484898 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484916 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\") pod \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\" (UID: \"2cb8de23-92b8-4bc0-8c35-16e51709a6a2\") " Oct 30 18:11:16.485099 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.484958 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\") pod \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\" (UID: \"ee05c9c6-d540-496c-8fc3-c1cdf5c88585\") " Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485126 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485174 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485202 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485366 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485481 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485496 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485510 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485522 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.485554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.485455 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.487087 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487061 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "kube-api-access-77wcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:16.487474 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487433 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.487474 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487438 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "kube-api-access-zxkfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:16.487474 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487443 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs" pod "2cb8de23-92b8-4bc0-8c35-16e51709a6a2" (UID: "2cb8de23-92b8-4bc0-8c35-16e51709a6a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:16.487636 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487564 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:11:16.487636 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.487575 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs" pod "ee05c9c6-d540-496c-8fc3-c1cdf5c88585" (UID: "ee05c9c6-d540-496c-8fc3-c1cdf5c88585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:16.586090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586064 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586088 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586099 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkfw\" (UniqueName: \"kubernetes.io/projected/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-kube-api-access-zxkfw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586107 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb8de23-92b8-4bc0-8c35-16e51709a6a2-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586116 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586123 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wcw\" (UniqueName: \"kubernetes.io/projected/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-kube-api-access-77wcw\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:16.586286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:16.586131 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee05c9c6-d540-496c-8fc3-c1cdf5c88585-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:17.222661 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.222628 4622 generic.go:356] "Generic (PLEG): container finished" podID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerID="83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8" exitCode=0 Oct 30 18:11:17.223095 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.222663 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" event={"ID":"ee05c9c6-d540-496c-8fc3-c1cdf5c88585","Type":"ContainerDied","Data":"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8"} Oct 30 18:11:17.223095 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.222691 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" Oct 30 18:11:17.223095 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.222716 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7" event={"ID":"ee05c9c6-d540-496c-8fc3-c1cdf5c88585","Type":"ContainerDied","Data":"47305bfd69d08e989e802f92138a52ac165f92dfa991fdd32fa8c1ba449390bf"} Oct 30 18:11:17.223095 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.222740 4622 scope.go:122] "RemoveContainer" containerID="83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8" Oct 30 18:11:17.224242 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.224224 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" Oct 30 18:11:17.224367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.224277 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2" event={"ID":"2cb8de23-92b8-4bc0-8c35-16e51709a6a2","Type":"ContainerDied","Data":"01135e2fff25e7997779c8a03e52e20b2dba3630505fe6baea4221d4c23f93cc"} Oct 30 18:11:17.232595 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.232576 4622 scope.go:122] "RemoveContainer" containerID="83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8" Oct 30 18:11:17.232944 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:17.232924 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8\": container with ID starting with 83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8 not found: ID does not exist" containerID="83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8" Oct 30 18:11:17.233011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.232956 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8"} err="failed to get container status \"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8\": rpc error: code = NotFound desc = could not find container \"83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8\": container with ID starting with 83ea2fcf487a21c7b53a606dd66503988dbf192647827a08a603845588a147c8 not found: ID does not exist" Oct 30 18:11:17.233011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.232977 4622 scope.go:122] "RemoveContainer" containerID="782f385df7c343f199284c11237e5215e3aa882c883cb40b23494c633c8ee4b6" Oct 30 18:11:17.269072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.269048 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 18:11:17.279357 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.279338 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/premium-w4-0e7403-kserve-78cdfdbfc7-v4wb7"] Oct 30 18:11:17.309220 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.309195 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 18:11:17.313084 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:17.313063 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0e7403/distinct-2-w4-0e7403-kserve-7f7d6595ff-62qp2"] Oct 30 18:11:18.372081 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:18.372050 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" path="/var/lib/kubelet/pods/2cb8de23-92b8-4bc0-8c35-16e51709a6a2/volumes" Oct 30 18:11:18.372480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:18.372459 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" path="/var/lib/kubelet/pods/ee05c9c6-d540-496c-8fc3-c1cdf5c88585/volumes" Oct 30 18:11:23.457075 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.457032 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:11:23.532893 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.532852 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 18:11:23.533150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.533116 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" podUID="a2748d16-0769-42cd-8ac8-3999a15c6b19" containerName="maas-api" containerID="cri-o://2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9" gracePeriod=30 Oct 30 18:11:23.782426 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.782392 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 18:11:23.945340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945306 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\") pod \"a2748d16-0769-42cd-8ac8-3999a15c6b19\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " Oct 30 18:11:23.945544 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945372 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\") pod \"a2748d16-0769-42cd-8ac8-3999a15c6b19\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " Oct 30 18:11:23.945544 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945401 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\" (UniqueName: \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\") pod \"a2748d16-0769-42cd-8ac8-3999a15c6b19\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " Oct 30 18:11:23.945544 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945496 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\") pod \"a2748d16-0769-42cd-8ac8-3999a15c6b19\" (UID: \"a2748d16-0769-42cd-8ac8-3999a15c6b19\") " Oct 30 18:11:23.945713 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945654 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca" pod "a2748d16-0769-42cd-8ac8-3999a15c6b19" (UID: "a2748d16-0769-42cd-8ac8-3999a15c6b19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:23.945801 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.945785 4622 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2748d16-0769-42cd-8ac8-3999a15c6b19-openshift-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:23.947547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.947517 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs" pod "a2748d16-0769-42cd-8ac8-3999a15c6b19" (UID: "a2748d16-0769-42cd-8ac8-3999a15c6b19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:23.947547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.947538 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg" pod "a2748d16-0769-42cd-8ac8-3999a15c6b19" (UID: "a2748d16-0769-42cd-8ac8-3999a15c6b19"). InnerVolumeSpecName "kube-api-access-57tqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:23.947704 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:23.947598 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls" pod "a2748d16-0769-42cd-8ac8-3999a15c6b19" (UID: "a2748d16-0769-42cd-8ac8-3999a15c6b19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:24.046185 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.046146 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-maas-api-tls\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:24.046185 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.046188 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57tqg\" (UniqueName: \"kubernetes.io/projected/a2748d16-0769-42cd-8ac8-3999a15c6b19-kube-api-access-57tqg\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:24.046408 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.046200 4622 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2748d16-0769-42cd-8ac8-3999a15c6b19-metrics-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:24.264047 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.264016 4622 generic.go:356] "Generic (PLEG): container finished" podID="a2748d16-0769-42cd-8ac8-3999a15c6b19" containerID="2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9" exitCode=0 Oct 30 18:11:24.264208 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.264061 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" Oct 30 18:11:24.264208 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.264102 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" event={"ID":"a2748d16-0769-42cd-8ac8-3999a15c6b19","Type":"ContainerDied","Data":"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9"} Oct 30 18:11:24.264208 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.264150 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8" event={"ID":"a2748d16-0769-42cd-8ac8-3999a15c6b19","Type":"ContainerDied","Data":"09d5c4d3efd1dd239529175c4f095feb01079634514b840d67d9a0ee9e315003"} Oct 30 18:11:24.264208 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.264173 4622 scope.go:122] "RemoveContainer" containerID="2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9" Oct 30 18:11:24.273996 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.273977 4622 scope.go:122] "RemoveContainer" containerID="2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9" Oct 30 18:11:24.274250 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:24.274230 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9\": container with ID starting with 2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9 not found: ID does not exist" containerID="2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9" Oct 30 18:11:24.274435 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.274257 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9"} err="failed to get container status \"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9\": rpc error: code = NotFound desc = could not find container \"2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9\": container with ID starting with 2b77c6b8ff40bdf530d9190770a043a5296347c04c2158a9c8b4e56fb05b42c9 not found: ID does not exist" Oct 30 18:11:24.310951 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.310898 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 18:11:24.321067 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.321047 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0e7403-f677f889c-f8zm8"] Oct 30 18:11:24.372889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:24.372865 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2748d16-0769-42cd-8ac8-3999a15c6b19" path="/var/lib/kubelet/pods/a2748d16-0769-42cd-8ac8-3999a15c6b19/volumes" Oct 30 18:11:24.534172 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:24.534143 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:11:32.010231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.010201 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 18:11:32.112406 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.112378 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\") pod \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " Oct 30 18:11:32.112582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.112422 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\" (UniqueName: \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\") pod \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\" (UID: \"dc8551c6-73d7-40a8-ba0e-4722339d7a9c\") " Oct 30 18:11:32.112825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.112800 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume" pod "dc8551c6-73d7-40a8-ba0e-4722339d7a9c" (UID: "dc8551c6-73d7-40a8-ba0e-4722339d7a9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:32.114526 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.114503 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9" pod "dc8551c6-73d7-40a8-ba0e-4722339d7a9c" (UID: "dc8551c6-73d7-40a8-ba0e-4722339d7a9c"). InnerVolumeSpecName "kube-api-access-qxkl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:32.213253 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.213223 4622 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-plugins-config-volume\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:32.213253 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.213248 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkl9\" (UniqueName: \"kubernetes.io/projected/dc8551c6-73d7-40a8-ba0e-4722339d7a9c-kube-api-access-qxkl9\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:11:32.311003 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.310972 4622 generic.go:356] "Generic (PLEG): container finished" podID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerID="80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0" exitCode=0 Oct 30 18:11:32.311177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.311025 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" Oct 30 18:11:32.311177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.311053 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" event={"ID":"dc8551c6-73d7-40a8-ba0e-4722339d7a9c","Type":"ContainerDied","Data":"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0"} Oct 30 18:11:32.311177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.311103 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7" event={"ID":"dc8551c6-73d7-40a8-ba0e-4722339d7a9c","Type":"ContainerDied","Data":"6619be461765c0f1a63e709a095d64f44231038e1afa318394c61aac92de8e88"} Oct 30 18:11:32.311177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.311127 4622 scope.go:122] "RemoveContainer" containerID="80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0" Oct 30 18:11:32.320583 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.320566 4622 scope.go:122] "RemoveContainer" containerID="80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0" Oct 30 18:11:32.320852 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:32.320834 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0\": container with ID starting with 80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0 not found: ID does not exist" containerID="80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0" Oct 30 18:11:32.320918 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.320865 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0"} err="failed to get container status \"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0\": rpc error: code = NotFound desc = could not find container \"80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0\": container with ID starting with 80657a4a65e436ac5a96d7960c1631e3b90e351893c48b6ac1e2508761ae67d0 not found: ID does not exist" Oct 30 18:11:32.357557 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.357532 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 18:11:32.372987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:32.372943 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0e7403-c4bbc55b7-xqzj7"] Oct 30 18:11:34.372247 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:34.372212 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" path="/var/lib/kubelet/pods/dc8551c6-73d7-40a8-ba0e-4722339d7a9c/volumes" Oct 30 18:11:36.033997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:36.033961 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k"] Oct 30 18:11:36.044646 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:36.044619 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90683a54-zbl7k"] Oct 30 18:11:36.372217 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:36.372128 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4f2cac-7bca-4af6-8137-721d7dff2276" path="/var/lib/kubelet/pods/1a4f2cac-7bca-4af6-8137-721d7dff2276/volumes" Oct 30 18:11:50.185499 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185465 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185830 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185840 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185861 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerName="discovery" Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185866 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerName="discovery" Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185875 4622 cpu_manager.go:414] "containerMap: removing container" podUID="564b66d8-b77c-4fb2-b34a-69e15936452e" containerName="payload-processing" Oct 30 18:11:50.185889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185880 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564b66d8-b77c-4fb2-b34a-69e15936452e" containerName="payload-processing" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185893 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185898 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185905 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185910 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185920 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" containerName="revoke-keys" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185925 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" containerName="revoke-keys" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185931 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185936 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185943 4622 cpu_manager.go:414] "containerMap: removing container" podUID="a2748d16-0769-42cd-8ac8-3999a15c6b19" containerName="maas-api" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.185948 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2748d16-0769-42cd-8ac8-3999a15c6b19" containerName="maas-api" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186004 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cf8876-b173-4519-8887-e9acf5c4005b" containerName="discovery" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186011 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="564b66d8-b77c-4fb2-b34a-69e15936452e" containerName="payload-processing" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186019 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31ffeb5-1de8-4da9-9140-5ed5cd104eb8" containerName="istio-proxy" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186025 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb8de23-92b8-4bc0-8c35-16e51709a6a2" containerName="main" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186032 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7e2fd4-3c4c-4cb8-99b0-fe6c8a595d35" containerName="revoke-keys" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186037 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8551c6-73d7-40a8-ba0e-4722339d7a9c" containerName="payload-pre-processing" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186045 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2748d16-0769-42cd-8ac8-3999a15c6b19" containerName="maas-api" Oct 30 18:11:50.186093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.186052 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee05c9c6-d540-496c-8fc3-c1cdf5c88585" containerName="main" Oct 30 18:11:50.189073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.189055 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.204936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.204912 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mccnk\"" Oct 30 18:11:50.217689 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.217666 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:11:50.268906 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.268875 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269015 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.268915 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269015 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.268940 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269143 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269044 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269182 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269137 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269182 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269160 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269181 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269208 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.269347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.269292 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cflq\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373020 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.372989 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cflq\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373062 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373096 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373133 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373158 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373203 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373232 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373278 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373305 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373662 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373637 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373728 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.373935 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.373911 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.374060 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.374041 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.374119 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.374082 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.375816 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.375795 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.375909 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.375864 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.394197 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.394177 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.394464 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.394445 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cflq\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\") pod \"maas-default-gateway-openshift-default-7df854bf67-4l8zv\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.502425 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.502357 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:50.810215 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.810188 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:11:50.811708 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:11:50.811678 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e692f06_e611_4656_8edc_6de03b31b126.slice/crio-030b157e7551c419497f566153aed72c8e72d6fd4fce78ce9cfad5212be412eb WatchSource:0}: Error finding container 030b157e7551c419497f566153aed72c8e72d6fd4fce78ce9cfad5212be412eb: Status 404 returned error can't find the container with id 030b157e7551c419497f566153aed72c8e72d6fd4fce78ce9cfad5212be412eb Oct 30 18:11:50.813724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.813696 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:50.813788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.813758 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:50.813829 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:50.813790 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:51.411597 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.411562 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" event={"ID":"4e692f06-e611-4656-8edc-6de03b31b126","Type":"ContainerStarted","Data":"38a6d4a1ca5aa1510b6aab8bb17a89e7a82ec820f49eb2a5e9aedc7cb721399a"} Oct 30 18:11:51.411991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.411616 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" event={"ID":"4e692f06-e611-4656-8edc-6de03b31b126","Type":"ContainerStarted","Data":"030b157e7551c419497f566153aed72c8e72d6fd4fce78ce9cfad5212be412eb"} Oct 30 18:11:51.477924 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.477866 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podStartSLOduration=1.477851583 podStartE2EDuration="1.477851583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:51.465691168 +0000 UTC m=+3539.243163124" watchObservedRunningTime="2026-10-30 18:11:51.477851583 +0000 UTC m=+3539.255323537" Oct 30 18:11:51.502479 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.502436 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:51.504052 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.504028 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:51.504106 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:51.504084 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:52.502978 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:52.502948 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:52.503384 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:52.503001 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:53.027326 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:53.027287 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh"] Oct 30 18:11:53.037626 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:53.037596 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c2f19f3-7g2sh"] Oct 30 18:11:53.503011 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:53.502973 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:53.503441 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:53.503026 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:54.372402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:54.372370 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb60840-9ff0-4fb1-ac0d-b4e0886868ec" path="/var/lib/kubelet/pods/0cb60840-9ff0-4fb1-ac0d-b4e0886868ec/volumes" Oct 30 18:11:54.502922 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:54.502884 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:54.503100 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:54.502940 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:54.532853 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:11:54.532818 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:11:55.503570 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:55.503480 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:55.503570 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:55.503536 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:56.503353 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:56.503316 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.503539 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:56.503382 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:57.503628 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:57.503596 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.504052 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:57.503647 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 18:11:58.983450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:58.983412 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:58.983941 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:58.983906 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:11:58.984702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:11:58.984677 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:12:05.246729 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.246691 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:12:05.250368 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.250347 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.276645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.276618 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:12:05.302585 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302556 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302593 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302611 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302792 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302755 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302792 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302782 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302851 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302902 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302873 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302902 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302892 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.302980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.302964 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9zk\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403755 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403722 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403755 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403760 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403777 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403817 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403834 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403876 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403911 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403933 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.403984 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.403975 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9zk\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.404405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.404239 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.404405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.404306 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.404485 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.404405 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.404672 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.404650 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.404731 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.404672 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.406209 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.406184 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.406523 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.406505 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.425329 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.425305 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.425516 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.425495 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9zk\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\") pod \"maas-default-gateway-openshift-default-7df854bf67-2kfst\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.564543 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.564514 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:05.873647 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.873620 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:12:05.875708 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:12:05.875681 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb93d64_c87c_41fb_9ecc_909def18795c.slice/crio-dfab09257f6450f1c12391fa6c7093340b5d9fb3689e667dbbd86ba73ddab9b9 WatchSource:0}: Error finding container dfab09257f6450f1c12391fa6c7093340b5d9fb3689e667dbbd86ba73ddab9b9: Status 404 returned error can't find the container with id dfab09257f6450f1c12391fa6c7093340b5d9fb3689e667dbbd86ba73ddab9b9 Oct 30 18:12:05.877889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.877854 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:05.877960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.877920 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:05.877960 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:05.877950 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:06.488757 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.488723 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" event={"ID":"8bb93d64-c87c-41fb-9ecc-909def18795c","Type":"ContainerStarted","Data":"bd4d433490aa4a478c40ba00fbfff870284d0f1204591ccd51f92c148cc6be7b"} Oct 30 18:12:06.489160 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.488776 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" event={"ID":"8bb93d64-c87c-41fb-9ecc-909def18795c","Type":"ContainerStarted","Data":"dfab09257f6450f1c12391fa6c7093340b5d9fb3689e667dbbd86ba73ddab9b9"} Oct 30 18:12:06.567487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.566552 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:06.568351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.568318 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:06.568611 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.568575 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:06.573073 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:06.573018 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podStartSLOduration=1.5730028520000001 podStartE2EDuration="1.573002852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:06.568650015 +0000 UTC m=+3554.346121973" watchObservedRunningTime="2026-10-30 18:12:06.573002852 +0000 UTC m=+3554.350474809" Oct 30 18:12:07.565781 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:07.565741 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:07.566184 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:07.565823 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:08.565738 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:08.565695 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:08.565959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:08.565857 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:09.565669 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:09.565629 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:09.565860 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:09.565696 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:10.565634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:10.565593 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:10.566036 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:10.565649 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:11.565692 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:11.565653 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:11.566214 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:11.565705 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:12.565732 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:12.565692 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 18:12:12.566114 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:12.565750 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 18:12:14.025321 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:14.025282 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:14.025820 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:14.025667 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:14.026575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:14.026555 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:12:24.533294 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:12:24.533233 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:12:54.532835 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:12:54.532784 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:12:55.203454 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:55.203408 4622 scope.go:122] "RemoveContainer" containerID="a210d825300779d56bcbb2a837d47aa6898af58675f499a6ea48135fd876420e" Oct 30 18:12:55.213294 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:12:55.213246 4622 scope.go:122] "RemoveContainer" containerID="8a9df2be3a07b73952841cf68577c615a32b661ae9ff8f51ba833efb2ed1cab2" Oct 30 18:13:05.426712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.426673 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t"] Oct 30 18:13:05.431184 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.431158 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.461246 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.461216 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t"] Oct 30 18:13:05.537679 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537651 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537817 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537688 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537817 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537716 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537817 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537764 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537854 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537888 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537916 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rxxg\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-kube-api-access-5rxxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.537974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.537941 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.538187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.538069 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.638972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.638948 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639089 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.638994 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639089 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639015 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639046 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639531 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639632 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639603 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639699 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639665 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639699 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639666 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639827 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639718 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rxxg\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-kube-api-access-5rxxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639827 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639748 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.639827 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.639754 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.640071 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.640052 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.640571 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.640546 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.640669 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.640566 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.647973 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.642396 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.648602 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.648583 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.661118 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.661090 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.661385 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.661368 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rxxg\" (UniqueName: \"kubernetes.io/projected/a7411194-34e7-4d6e-8415-efb9c6e32fc1-kube-api-access-5rxxg\") pod \"maas-default-gateway-openshift-default-7df854bf67-b7p7t\" (UID: \"a7411194-34e7-4d6e-8415-efb9c6e32fc1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:05.744743 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:05.744690 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:06.059227 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.059198 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t"] Oct 30 18:13:06.062343 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:13:06.062312 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7411194_34e7_4d6e_8415_efb9c6e32fc1.slice/crio-9a1929f36815cbbfd09b6f0483245006478a810fded9974bceae6ff0beeb1a4f WatchSource:0}: Error finding container 9a1929f36815cbbfd09b6f0483245006478a810fded9974bceae6ff0beeb1a4f: Status 404 returned error can't find the container with id 9a1929f36815cbbfd09b6f0483245006478a810fded9974bceae6ff0beeb1a4f Oct 30 18:13:06.064336 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.064298 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:06.064428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.064366 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:06.064428 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.064394 4622 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:06.783484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.783444 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" event={"ID":"a7411194-34e7-4d6e-8415-efb9c6e32fc1","Type":"ContainerStarted","Data":"9dafb24ec6e0d6396601f978181b42a9194fc84e1b83e594d048729f9903d706"} Oct 30 18:13:06.783914 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.783500 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" event={"ID":"a7411194-34e7-4d6e-8415-efb9c6e32fc1","Type":"ContainerStarted","Data":"9a1929f36815cbbfd09b6f0483245006478a810fded9974bceae6ff0beeb1a4f"} Oct 30 18:13:06.864287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:06.863744 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podStartSLOduration=1.8637111 podStartE2EDuration="1.8637111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:06.835531871 +0000 UTC m=+3614.613003827" watchObservedRunningTime="2026-10-30 18:13:06.8637111 +0000 UTC m=+3614.641183056" Oct 30 18:13:07.763286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:07.762654 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:07.763286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:07.762776 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:07.763286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:07.762820 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:08.755287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:08.746604 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:08.755287 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:08.746667 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:09.753289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:09.752691 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:09.753289 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:09.752752 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:10.754299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:10.749656 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:10.754299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:10.749718 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:11.745642 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:11.745600 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:11.745917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:11.745887 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:12.745438 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:12.745397 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.745947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:12.745464 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 18:13:14.745830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:14.745792 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:14.746241 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:14.745851 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:15.787434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:15.787385 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b7p7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:15.787946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:15.787446 4622 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" podUID="a7411194-34e7-4d6e-8415-efb9c6e32fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:16.205202 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:16.205108 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:16.205473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:16.205450 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:16.206210 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:16.206187 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b7p7t" Oct 30 18:13:24.537718 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:13:24.537628 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:13:43.027560 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:43.027529 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2"] Oct 30 18:13:43.037789 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:43.037760 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ef1e68-z22x2"] Oct 30 18:13:44.374987 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:13:44.374563 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddbe6c86-7fa1-45f9-aa21-da4e59583f74" path="/var/lib/kubelet/pods/ddbe6c86-7fa1-45f9-aa21-da4e59583f74/volumes" Oct 30 18:13:54.533517 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:13:54.533478 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:14:24.533443 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:14:24.533407 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:14:54.533078 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:14:54.533038 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:14:55.245070 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:14:55.245041 4622 scope.go:122] "RemoveContainer" containerID="b29676dfd48a2a40106be21b750b8847716426489b34bd076c28b5f49b5fea56" Oct 30 18:15:11.039829 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.039794 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6"] Oct 30 18:15:11.043202 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.043186 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.056401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.056374 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:15:11.056401 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.056396 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:15:11.066840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.066820 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:11.067006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.066988 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:15:11.070052 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.070027 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6"] Oct 30 18:15:11.154164 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.154140 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.154306 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.154171 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knfqx\" (UniqueName: \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.154306 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.154206 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.255582 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.255556 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.255677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.255587 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knfqx\" (UniqueName: \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.255677 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.255616 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.256187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.256169 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.257867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.257849 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.277129 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.277088 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knfqx\" (UniqueName: \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\") pod \"maas-api-revoke-sub-ae594979-bdhm6\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.355513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.355459 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:11.648107 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:11.648068 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6"] Oct 30 18:15:11.650395 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:15:11.650367 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69984b6b_db03_44dd_88a0_432462b5ee76.slice/crio-311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af WatchSource:0}: Error finding container 311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af: Status 404 returned error can't find the container with id 311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af Oct 30 18:15:12.490318 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:12.490289 4622 generic.go:356] "Generic (PLEG): container finished" podID="69984b6b-db03-44dd-88a0-432462b5ee76" containerID="b310d124c51f70ba281e5e91a4898c7a78e5aec996b1cf8e780da371dc97cdc1" exitCode=0 Oct 30 18:15:12.490659 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:12.490332 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" event={"ID":"69984b6b-db03-44dd-88a0-432462b5ee76","Type":"ContainerDied","Data":"b310d124c51f70ba281e5e91a4898c7a78e5aec996b1cf8e780da371dc97cdc1"} Oct 30 18:15:12.490659 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:12.490383 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" event={"ID":"69984b6b-db03-44dd-88a0-432462b5ee76","Type":"ContainerStarted","Data":"311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af"} Oct 30 18:15:13.627155 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.627129 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:13.675488 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.675464 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\" (UniqueName: \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\") pod \"69984b6b-db03-44dd-88a0-432462b5ee76\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " Oct 30 18:15:13.675610 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.675558 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\") pod \"69984b6b-db03-44dd-88a0-432462b5ee76\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " Oct 30 18:15:13.675664 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.675625 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\") pod \"69984b6b-db03-44dd-88a0-432462b5ee76\" (UID: \"69984b6b-db03-44dd-88a0-432462b5ee76\") " Oct 30 18:15:13.676044 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.676018 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca" pod "69984b6b-db03-44dd-88a0-432462b5ee76" (UID: "69984b6b-db03-44dd-88a0-432462b5ee76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:13.677499 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.677472 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx" pod "69984b6b-db03-44dd-88a0-432462b5ee76" (UID: "69984b6b-db03-44dd-88a0-432462b5ee76"). InnerVolumeSpecName "kube-api-access-knfqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:13.677596 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.677567 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca" pod "69984b6b-db03-44dd-88a0-432462b5ee76" (UID: "69984b6b-db03-44dd-88a0-432462b5ee76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:13.777323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.777248 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:15:13.777323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.777293 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69984b6b-db03-44dd-88a0-432462b5ee76-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:15:13.777323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:13.777305 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knfqx\" (UniqueName: \"kubernetes.io/projected/69984b6b-db03-44dd-88a0-432462b5ee76-kube-api-access-knfqx\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:15:14.500669 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:14.500640 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" Oct 30 18:15:14.500669 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:14.500661 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6" event={"ID":"69984b6b-db03-44dd-88a0-432462b5ee76","Type":"ContainerDied","Data":"311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af"} Oct 30 18:15:14.500864 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:14.500704 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311032d8790d9fd938cca6dc8dab5c36923fe2617f53c9a646a1594a445534af" Oct 30 18:15:24.533123 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:15:24.533084 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:15:54.532284 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:15:54.532220 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:15:55.265163 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:15:55.265133 4622 scope.go:122] "RemoveContainer" containerID="c25dc4322c4c9da216bed8c38994d09f8325c96af2056f5496303f3f8fb94fe9" Oct 30 18:16:24.533006 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:16:24.532925 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:16:54.532362 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:16:54.532318 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:17:24.532800 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:17:24.532761 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:17:54.533246 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:17:54.533164 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:18:04.005975 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.005940 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:04.006461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.006337 4622 cpu_manager.go:414] "containerMap: removing container" podUID="69984b6b-db03-44dd-88a0-432462b5ee76" containerName="revoke-keys" Oct 30 18:18:04.006461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.006350 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69984b6b-db03-44dd-88a0-432462b5ee76" containerName="revoke-keys" Oct 30 18:18:04.006461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.006429 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="69984b6b-db03-44dd-88a0-432462b5ee76" containerName="revoke-keys" Oct 30 18:18:04.010947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.010909 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.025476 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.025452 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-05e969\"/\"kube-root-ca.crt\"" Oct 30 18:18:04.035119 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.035092 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-05e969\"/\"openshift-service-ca.crt\"" Oct 30 18:18:04.036169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.036142 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-05e969\"/\"simulator-single-main-05e969-kserve-self-signed-certs\"" Oct 30 18:18:04.036325 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.036166 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-05e969\"/\"default-dockercfg-xv928\"" Oct 30 18:18:04.037799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.037774 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:04.121842 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.121808 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzs22\" (UniqueName: \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.121970 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.121849 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.122027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.121983 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.122027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.122013 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.122111 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.122056 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.122149 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.122127 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223453 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223497 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223750 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223541 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223750 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223586 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223750 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223655 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzs22\" (UniqueName: \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.223750 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223683 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.224000 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.223943 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.224098 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.224077 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.224168 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.224146 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.225787 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.225761 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.226311 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.226290 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.246936 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.246908 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzs22\" (UniqueName: \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\") pod \"simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.330473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.330444 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:04.624895 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:04.624715 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:04.627226 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:18:04.627189 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa467038_b2dd_4534_a285_859d2756384c.slice/crio-96b93d9b66b8229a71138217bd6876f19ffa275fc453ad07308964cc944d0908 WatchSource:0}: Error finding container 96b93d9b66b8229a71138217bd6876f19ffa275fc453ad07308964cc944d0908: Status 404 returned error can't find the container with id 96b93d9b66b8229a71138217bd6876f19ffa275fc453ad07308964cc944d0908 Oct 30 18:18:05.429005 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:05.428970 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" event={"ID":"aa467038-b2dd-4534-a285-859d2756384c","Type":"ContainerStarted","Data":"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9"} Oct 30 18:18:05.429523 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:05.429013 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" event={"ID":"aa467038-b2dd-4534-a285-859d2756384c","Type":"ContainerStarted","Data":"96b93d9b66b8229a71138217bd6876f19ffa275fc453ad07308964cc944d0908"} Oct 30 18:18:05.468920 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:05.468873 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" podStartSLOduration=2.4688607989999998 podStartE2EDuration="2.468860799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:05.466427748 +0000 UTC m=+3913.243899702" watchObservedRunningTime="2026-10-30 18:18:05.468860799 +0000 UTC m=+3913.246332808" Oct 30 18:18:06.330721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:06.330690 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:14.331102 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:14.331064 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:14.343690 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:14.343668 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:14.488539 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:14.488510 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:24.533359 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:18:24.533317 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:18:25.532549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.532509 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb"] Oct 30 18:18:25.537517 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.537492 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.552914 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.552887 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb"] Oct 30 18:18:25.560997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.560967 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:25.561105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.561018 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:25.561105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.561041 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:18:25.561105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.561068 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:25.613113 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.613086 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqnm\" (UniqueName: \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.613221 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.613144 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.613297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.613248 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.714552 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.714526 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqnm\" (UniqueName: \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.714657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.714587 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.714657 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.714620 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.715238 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.715214 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.716923 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.716898 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.735563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.735540 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqnm\" (UniqueName: \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\") pod \"maas-api-revoke-sub-31c750cc-mnmvb\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:25.850709 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:25.850647 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:26.146046 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:26.146019 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb"] Oct 30 18:18:26.148371 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:18:26.148335 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c048e62_fc80_43d3_bb85_f056e4ac527f.slice/crio-0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645 WatchSource:0}: Error finding container 0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645: Status 404 returned error can't find the container with id 0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645 Oct 30 18:18:26.548793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:26.548755 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" event={"ID":"0c048e62-fc80-43d3-bb85-f056e4ac527f","Type":"ContainerStarted","Data":"1db0c388715df32088b84fde1e223d04f662ea9f3bc7070a9a8ddfc14928e402"} Oct 30 18:18:26.549204 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:26.548807 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" event={"ID":"0c048e62-fc80-43d3-bb85-f056e4ac527f","Type":"ContainerStarted","Data":"0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645"} Oct 30 18:18:27.555728 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:27.555692 4622 generic.go:356] "Generic (PLEG): container finished" podID="0c048e62-fc80-43d3-bb85-f056e4ac527f" containerID="1db0c388715df32088b84fde1e223d04f662ea9f3bc7070a9a8ddfc14928e402" exitCode=0 Oct 30 18:18:27.556131 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:27.555783 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" event={"ID":"0c048e62-fc80-43d3-bb85-f056e4ac527f","Type":"ContainerDied","Data":"1db0c388715df32088b84fde1e223d04f662ea9f3bc7070a9a8ddfc14928e402"} Oct 30 18:18:28.690479 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.690452 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:28.742540 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.742514 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\") pod \"0c048e62-fc80-43d3-bb85-f056e4ac527f\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " Oct 30 18:18:28.742659 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.742557 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\") pod \"0c048e62-fc80-43d3-bb85-f056e4ac527f\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " Oct 30 18:18:28.742707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.742659 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\" (UniqueName: \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\") pod \"0c048e62-fc80-43d3-bb85-f056e4ac527f\" (UID: \"0c048e62-fc80-43d3-bb85-f056e4ac527f\") " Oct 30 18:18:28.742914 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.742893 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca" pod "0c048e62-fc80-43d3-bb85-f056e4ac527f" (UID: "0c048e62-fc80-43d3-bb85-f056e4ac527f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:28.742991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.742974 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.744591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.744567 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca" pod "0c048e62-fc80-43d3-bb85-f056e4ac527f" (UID: "0c048e62-fc80-43d3-bb85-f056e4ac527f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:28.744796 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.744773 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm" pod "0c048e62-fc80-43d3-bb85-f056e4ac527f" (UID: "0c048e62-fc80-43d3-bb85-f056e4ac527f"). InnerVolumeSpecName "kube-api-access-kzqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:28.843469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.843410 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzqnm\" (UniqueName: \"kubernetes.io/projected/0c048e62-fc80-43d3-bb85-f056e4ac527f-kube-api-access-kzqnm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:28.843469 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:28.843433 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c048e62-fc80-43d3-bb85-f056e4ac527f-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:29.567230 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:29.567200 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" Oct 30 18:18:29.567437 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:29.567199 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb" event={"ID":"0c048e62-fc80-43d3-bb85-f056e4ac527f","Type":"ContainerDied","Data":"0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645"} Oct 30 18:18:29.567437 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:29.567331 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0037a7be59cfd915b522fae0c2675306e6ef3a40bac3c5f5083d8dcf60b5e645" Oct 30 18:18:36.837315 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.837279 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt"] Oct 30 18:18:36.837780 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.837671 4622 cpu_manager.go:414] "containerMap: removing container" podUID="0c048e62-fc80-43d3-bb85-f056e4ac527f" containerName="revoke-keys" Oct 30 18:18:36.837780 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.837682 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c048e62-fc80-43d3-bb85-f056e4ac527f" containerName="revoke-keys" Oct 30 18:18:36.837780 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.837757 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c048e62-fc80-43d3-bb85-f056e4ac527f" containerName="revoke-keys" Oct 30 18:18:36.840854 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.840834 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:36.854051 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.854025 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:36.854162 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.854077 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:18:36.864451 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.864426 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:18:36.864591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.864570 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:18:36.866907 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.866886 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt"] Oct 30 18:18:36.906406 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.906364 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd259\" (UniqueName: \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:36.906523 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.906417 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:36.906589 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:36.906566 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.007298 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.007252 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.007421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.007319 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd259\" (UniqueName: \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.007421 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.007347 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.007976 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.007956 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.009514 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.009496 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.028795 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.028770 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd259\" (UniqueName: \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\") pod \"maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.154743 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.154691 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:37.450136 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.450105 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt"] Oct 30 18:18:37.452023 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:18:37.451992 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc64886_ce74_4555_bfad_12e7cd4732f4.slice/crio-7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8 WatchSource:0}: Error finding container 7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8: Status 404 returned error can't find the container with id 7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8 Oct 30 18:18:37.608789 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:37.608751 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" event={"ID":"ebc64886-ce74-4555-bfad-12e7cd4732f4","Type":"ContainerStarted","Data":"7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8"} Oct 30 18:18:38.614904 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:38.614869 4622 generic.go:356] "Generic (PLEG): container finished" podID="ebc64886-ce74-4555-bfad-12e7cd4732f4" containerID="9923f8d78c81fb8fe194caf51bdec9e122b35735f8722068da872d20fabe5045" exitCode=0 Oct 30 18:18:38.615286 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:38.614918 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" event={"ID":"ebc64886-ce74-4555-bfad-12e7cd4732f4","Type":"ContainerDied","Data":"9923f8d78c81fb8fe194caf51bdec9e122b35735f8722068da872d20fabe5045"} Oct 30 18:18:39.748756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.748726 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:39.832793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.832767 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\") pod \"ebc64886-ce74-4555-bfad-12e7cd4732f4\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " Oct 30 18:18:39.832951 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.832810 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\") pod \"ebc64886-ce74-4555-bfad-12e7cd4732f4\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " Oct 30 18:18:39.832951 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.832862 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\" (UniqueName: \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\") pod \"ebc64886-ce74-4555-bfad-12e7cd4732f4\" (UID: \"ebc64886-ce74-4555-bfad-12e7cd4732f4\") " Oct 30 18:18:39.833117 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.833097 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca" pod "ebc64886-ce74-4555-bfad-12e7cd4732f4" (UID: "ebc64886-ce74-4555-bfad-12e7cd4732f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:39.834968 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.834946 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca" pod "ebc64886-ce74-4555-bfad-12e7cd4732f4" (UID: "ebc64886-ce74-4555-bfad-12e7cd4732f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:39.835079 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.835058 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259" pod "ebc64886-ce74-4555-bfad-12e7cd4732f4" (UID: "ebc64886-ce74-4555-bfad-12e7cd4732f4"). InnerVolumeSpecName "kube-api-access-wd259". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:39.934124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.934069 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:39.934124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.934091 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebc64886-ce74-4555-bfad-12e7cd4732f4-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:39.934124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:39.934101 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd259\" (UniqueName: \"kubernetes.io/projected/ebc64886-ce74-4555-bfad-12e7cd4732f4-kube-api-access-wd259\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:40.627031 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:40.627001 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" Oct 30 18:18:40.627031 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:40.627013 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt" event={"ID":"ebc64886-ce74-4555-bfad-12e7cd4732f4","Type":"ContainerDied","Data":"7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8"} Oct 30 18:18:40.627235 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:40.627060 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a0b60ebac9edea3c2f5357d99eb6a7ae65d34918caf3f1e85b3327a2c763fc8" Oct 30 18:18:41.624703 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:41.624668 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:49.763622 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:49.763589 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt"] Oct 30 18:18:49.773834 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:49.773805 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-05e969-t4twt"] Oct 30 18:18:50.372955 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:50.372919 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc64886-ce74-4555-bfad-12e7cd4732f4" path="/var/lib/kubelet/pods/ebc64886-ce74-4555-bfad-12e7cd4732f4/volumes" Oct 30 18:18:54.533105 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:18:54.533061 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:18:56.644575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.644535 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" podUID="aa467038-b2dd-4534-a285-859d2756384c" containerName="main" containerID="cri-o://b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9" gracePeriod=45 Oct 30 18:18:56.897577 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.897520 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:56.978255 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978223 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978432 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978303 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978432 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978327 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978432 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978375 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978444 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978507 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\" (UniqueName: \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\") pod \"aa467038-b2dd-4534-a285-859d2756384c\" (UID: \"aa467038-b2dd-4534-a285-859d2756384c\") " Oct 30 18:18:56.978548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978520 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:56.978748 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978723 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:56.978898 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978760 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:56.978898 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.978826 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:56.980836 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.980814 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "kube-api-access-hzs22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:56.980947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.980872 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:18:56.980947 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:56.980894 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs" pod "aa467038-b2dd-4534-a285-859d2756384c" (UID: "aa467038-b2dd-4534-a285-859d2756384c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:57.079513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.079481 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:57.079513 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.079511 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:57.079708 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.079524 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzs22\" (UniqueName: \"kubernetes.io/projected/aa467038-b2dd-4534-a285-859d2756384c-kube-api-access-hzs22\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:57.079708 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.079534 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa467038-b2dd-4534-a285-859d2756384c-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:57.079708 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.079542 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa467038-b2dd-4534-a285-859d2756384c-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:18:57.714224 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.714188 4622 generic.go:356] "Generic (PLEG): container finished" podID="aa467038-b2dd-4534-a285-859d2756384c" containerID="b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9" exitCode=0 Oct 30 18:18:57.714720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.714236 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" Oct 30 18:18:57.714720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.714287 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" event={"ID":"aa467038-b2dd-4534-a285-859d2756384c","Type":"ContainerDied","Data":"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9"} Oct 30 18:18:57.714720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.714343 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9" event={"ID":"aa467038-b2dd-4534-a285-859d2756384c","Type":"ContainerDied","Data":"96b93d9b66b8229a71138217bd6876f19ffa275fc453ad07308964cc944d0908"} Oct 30 18:18:57.714720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.714368 4622 scope.go:122] "RemoveContainer" containerID="b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9" Oct 30 18:18:57.724372 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.724355 4622 scope.go:122] "RemoveContainer" containerID="b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9" Oct 30 18:18:57.724643 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:18:57.724624 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9\": container with ID starting with b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9 not found: ID does not exist" containerID="b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9" Oct 30 18:18:57.724710 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.724659 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9"} err="failed to get container status \"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9\": rpc error: code = NotFound desc = could not find container \"b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9\": container with ID starting with b5dea922e56d091d37a2fd03a8207ffb41cc99ca8249481d2e2faa58538fe8b9 not found: ID does not exist" Oct 30 18:18:57.762213 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.762180 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:57.772004 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:57.771977 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-05e969/simulator-single-main-05e969-kserve-5cfbd8894b-f5rz9"] Oct 30 18:18:58.372472 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:18:58.372440 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa467038-b2dd-4534-a285-859d2756384c" path="/var/lib/kubelet/pods/aa467038-b2dd-4534-a285-859d2756384c/volumes" Oct 30 18:19:24.532660 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:19:24.532584 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:19:54.532567 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:19:54.532525 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:20:14.027792 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:20:14.027756 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6"] Oct 30 18:20:14.037870 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:20:14.037839 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bdhm6"] Oct 30 18:20:14.372466 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:20:14.372436 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69984b6b-db03-44dd-88a0-432462b5ee76" path="/var/lib/kubelet/pods/69984b6b-db03-44dd-88a0-432462b5ee76/volumes" Oct 30 18:20:24.532930 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:20:24.532891 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:20:54.532781 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:20:54.532703 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:21:24.533564 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:21:24.533528 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:21:54.532522 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:21:54.532487 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:21:55.331616 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:21:55.331583 4622 scope.go:122] "RemoveContainer" containerID="b310d124c51f70ba281e5e91a4898c7a78e5aec996b1cf8e780da371dc97cdc1" Oct 30 18:22:05.800385 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800351 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:22:05.800850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800741 4622 cpu_manager.go:414] "containerMap: removing container" podUID="ebc64886-ce74-4555-bfad-12e7cd4732f4" containerName="revoke-keys" Oct 30 18:22:05.800850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800753 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc64886-ce74-4555-bfad-12e7cd4732f4" containerName="revoke-keys" Oct 30 18:22:05.800850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800784 4622 cpu_manager.go:414] "containerMap: removing container" podUID="aa467038-b2dd-4534-a285-859d2756384c" containerName="main" Oct 30 18:22:05.800850 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800790 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa467038-b2dd-4534-a285-859d2756384c" containerName="main" Oct 30 18:22:05.800991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800862 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc64886-ce74-4555-bfad-12e7cd4732f4" containerName="revoke-keys" Oct 30 18:22:05.800991 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.800876 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa467038-b2dd-4534-a285-859d2756384c" containerName="main" Oct 30 18:22:05.805370 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.805346 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.820526 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.820495 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-67382b\"/\"simulator-single-main-67382b-kserve-self-signed-certs\"" Oct 30 18:22:05.831377 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.831341 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-67382b\"/\"openshift-service-ca.crt\"" Oct 30 18:22:05.831545 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.831427 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-67382b\"/\"kube-root-ca.crt\"" Oct 30 18:22:05.833946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.833923 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:22:05.844672 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844651 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.844793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844699 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.844793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844726 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.844886 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844804 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.844939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844901 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.844998 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.844964 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqf4\" (UniqueName: \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.917645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.917620 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-67382b\"/\"default-dockercfg-phdkq\"" Oct 30 18:22:05.946124 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946095 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946134 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqf4\" (UniqueName: \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946180 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946215 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946283 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946239 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946528 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946290 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946661 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946632 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946788 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946760 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.946860 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.946793 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.948585 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.948559 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.948830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.948812 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:05.967468 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:05.967441 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqf4\" (UniqueName: \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\") pod \"simulator-single-main-67382b-kserve-78654646d6-pgr5c\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:06.118147 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:06.118071 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:06.412821 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:06.412705 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:22:06.415029 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:22:06.415003 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf1da3e1_9470_4fde_a508_3bf86bf1c38a.slice/crio-c04e2d00632ecc6111e6ea5b5f15ed459e0c79c408a795f066982174e2818b34 WatchSource:0}: Error finding container c04e2d00632ecc6111e6ea5b5f15ed459e0c79c408a795f066982174e2818b34: Status 404 returned error can't find the container with id c04e2d00632ecc6111e6ea5b5f15ed459e0c79c408a795f066982174e2818b34 Oct 30 18:22:06.732653 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:06.732494 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" event={"ID":"af1da3e1-9470-4fde-a508-3bf86bf1c38a","Type":"ContainerStarted","Data":"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6"} Oct 30 18:22:06.732653 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:06.732565 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" event={"ID":"af1da3e1-9470-4fde-a508-3bf86bf1c38a","Type":"ContainerStarted","Data":"c04e2d00632ecc6111e6ea5b5f15ed459e0c79c408a795f066982174e2818b34"} Oct 30 18:22:06.772178 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:06.772123 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" podStartSLOduration=1.7721098830000002 podStartE2EDuration="1.772109883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:06.769570632 +0000 UTC m=+4154.547042587" watchObservedRunningTime="2026-10-30 18:22:06.772109883 +0000 UTC m=+4154.549581838" Oct 30 18:22:07.119226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:07.119196 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:16.119132 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:16.119095 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:16.131504 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:16.131477 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:16.806751 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:16.806718 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:22:24.533138 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:22:24.533056 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:22:34.870672 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.870632 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g"] Oct 30 18:22:34.874118 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.874098 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:34.887418 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.887394 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:34.897974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.897947 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:34.898090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.897976 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:34.898398 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.898376 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:22:34.901096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.901073 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g"] Oct 30 18:22:34.993195 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.993171 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:34.993312 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.993204 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:34.993371 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:34.993331 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmjh\" (UniqueName: \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.093771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.093741 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.093901 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.093775 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.093901 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.093810 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmjh\" (UniqueName: \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.094418 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.094398 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.095911 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.095890 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.115954 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.115929 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmjh\" (UniqueName: \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\") pod \"maas-api-revoke-sub-c334f6dd-nkm2g\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.187614 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.187559 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:35.481520 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.481492 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g"] Oct 30 18:22:35.484216 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:22:35.484184 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aceb7a9_3572_445b_aeb3_ca6c875698c3.slice/crio-2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52 WatchSource:0}: Error finding container 2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52: Status 404 returned error can't find the container with id 2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52 Oct 30 18:22:35.898817 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.898782 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" event={"ID":"4aceb7a9-3572-445b-aeb3-ca6c875698c3","Type":"ContainerStarted","Data":"6bbbd9e831781328c2809fab1536f7129f6f2825b0251dde536f1862b2888d50"} Oct 30 18:22:35.899186 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.898864 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" event={"ID":"4aceb7a9-3572-445b-aeb3-ca6c875698c3","Type":"ContainerStarted","Data":"2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52"} Oct 30 18:22:35.936226 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:35.936173 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" podStartSLOduration=1.936157137 podStartE2EDuration="1.936157137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:35.934902759 +0000 UTC m=+4183.712374715" watchObservedRunningTime="2026-10-30 18:22:35.936157137 +0000 UTC m=+4183.713629093" Oct 30 18:22:36.905178 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:36.905141 4622 generic.go:356] "Generic (PLEG): container finished" podID="4aceb7a9-3572-445b-aeb3-ca6c875698c3" containerID="6bbbd9e831781328c2809fab1536f7129f6f2825b0251dde536f1862b2888d50" exitCode=0 Oct 30 18:22:36.905679 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:36.905195 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" event={"ID":"4aceb7a9-3572-445b-aeb3-ca6c875698c3","Type":"ContainerDied","Data":"6bbbd9e831781328c2809fab1536f7129f6f2825b0251dde536f1862b2888d50"} Oct 30 18:22:38.043481 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.043451 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:38.222917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.222841 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\") pod \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " Oct 30 18:22:38.222917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.222886 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\" (UniqueName: \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\") pod \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " Oct 30 18:22:38.223150 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.223001 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\") pod \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\" (UID: \"4aceb7a9-3572-445b-aeb3-ca6c875698c3\") " Oct 30 18:22:38.223236 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.223214 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca" pod "4aceb7a9-3572-445b-aeb3-ca6c875698c3" (UID: "4aceb7a9-3572-445b-aeb3-ca6c875698c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:38.223429 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.223406 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:38.225035 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.225013 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca" pod "4aceb7a9-3572-445b-aeb3-ca6c875698c3" (UID: "4aceb7a9-3572-445b-aeb3-ca6c875698c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:38.225115 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.225094 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh" pod "4aceb7a9-3572-445b-aeb3-ca6c875698c3" (UID: "4aceb7a9-3572-445b-aeb3-ca6c875698c3"). InnerVolumeSpecName "kube-api-access-pqmjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:38.324639 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.324615 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmjh\" (UniqueName: \"kubernetes.io/projected/4aceb7a9-3572-445b-aeb3-ca6c875698c3-kube-api-access-pqmjh\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:38.324639 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.324638 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aceb7a9-3572-445b-aeb3-ca6c875698c3-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:38.916218 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.916188 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" Oct 30 18:22:38.916393 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.916187 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g" event={"ID":"4aceb7a9-3572-445b-aeb3-ca6c875698c3","Type":"ContainerDied","Data":"2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52"} Oct 30 18:22:38.916393 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:38.916311 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2785a8cbe5d02a316a03eedf2b099c864c252522b583417efe19f541e2a10d52" Oct 30 18:22:45.889159 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.889115 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz"] Oct 30 18:22:45.889718 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.889696 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4aceb7a9-3572-445b-aeb3-ca6c875698c3" containerName="revoke-keys" Oct 30 18:22:45.889810 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.889721 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aceb7a9-3572-445b-aeb3-ca6c875698c3" containerName="revoke-keys" Oct 30 18:22:45.889893 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.889880 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aceb7a9-3572-445b-aeb3-ca6c875698c3" containerName="revoke-keys" Oct 30 18:22:45.893084 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.893061 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:45.909047 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.909021 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:45.919771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.919745 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:45.919887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.919786 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:45.919887 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.919806 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:22:45.922183 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.922162 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz"] Oct 30 18:22:45.988627 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.988595 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:45.988802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.988657 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv824\" (UniqueName: \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:45.988802 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:45.988718 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.089215 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.089181 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.089392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.089239 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv824\" (UniqueName: \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.089392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.089255 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.089897 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.089864 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.091611 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.091590 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.111029 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.110999 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv824\" (UniqueName: \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\") pod \"maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.206424 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.206347 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:46.500791 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.500764 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz"] Oct 30 18:22:46.506184 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:22:46.506152 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde14f02_d51c_405a_96b8_05a868bcc9e4.slice/crio-375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad WatchSource:0}: Error finding container 375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad: Status 404 returned error can't find the container with id 375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad Oct 30 18:22:46.957939 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.957897 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" event={"ID":"dde14f02-d51c-405a-96b8-05a868bcc9e4","Type":"ContainerStarted","Data":"11f29f2b323209a59b2606250d55f7ef05f998d4e0a7f17fab0b121c1e3e2474"} Oct 30 18:22:46.958297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.957967 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" event={"ID":"dde14f02-d51c-405a-96b8-05a868bcc9e4","Type":"ContainerStarted","Data":"375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad"} Oct 30 18:22:46.995309 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:46.995249 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" podStartSLOduration=1.995236395 podStartE2EDuration="1.995236395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:46.99496446 +0000 UTC m=+4194.772436417" watchObservedRunningTime="2026-10-30 18:22:46.995236395 +0000 UTC m=+4194.772708349" Oct 30 18:22:47.964954 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:47.964871 4622 generic.go:356] "Generic (PLEG): container finished" podID="dde14f02-d51c-405a-96b8-05a868bcc9e4" containerID="11f29f2b323209a59b2606250d55f7ef05f998d4e0a7f17fab0b121c1e3e2474" exitCode=0 Oct 30 18:22:47.964954 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:47.964915 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" event={"ID":"dde14f02-d51c-405a-96b8-05a868bcc9e4","Type":"ContainerDied","Data":"11f29f2b323209a59b2606250d55f7ef05f998d4e0a7f17fab0b121c1e3e2474"} Oct 30 18:22:49.103547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.103520 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:49.115867 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.115846 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\" (UniqueName: \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\") pod \"dde14f02-d51c-405a-96b8-05a868bcc9e4\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " Oct 30 18:22:49.115965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.115889 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\") pod \"dde14f02-d51c-405a-96b8-05a868bcc9e4\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " Oct 30 18:22:49.115965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.115936 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\") pod \"dde14f02-d51c-405a-96b8-05a868bcc9e4\" (UID: \"dde14f02-d51c-405a-96b8-05a868bcc9e4\") " Oct 30 18:22:49.116278 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.116242 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca" pod "dde14f02-d51c-405a-96b8-05a868bcc9e4" (UID: "dde14f02-d51c-405a-96b8-05a868bcc9e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:49.116404 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.116373 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:49.117889 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.117868 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824" pod "dde14f02-d51c-405a-96b8-05a868bcc9e4" (UID: "dde14f02-d51c-405a-96b8-05a868bcc9e4"). InnerVolumeSpecName "kube-api-access-wv824". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:49.118040 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.118020 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca" pod "dde14f02-d51c-405a-96b8-05a868bcc9e4" (UID: "dde14f02-d51c-405a-96b8-05a868bcc9e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:49.217830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.217787 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dde14f02-d51c-405a-96b8-05a868bcc9e4-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:49.217830 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.217825 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv824\" (UniqueName: \"kubernetes.io/projected/dde14f02-d51c-405a-96b8-05a868bcc9e4-kube-api-access-wv824\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:22:49.977656 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.977627 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" Oct 30 18:22:49.977835 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.977633 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz" event={"ID":"dde14f02-d51c-405a-96b8-05a868bcc9e4","Type":"ContainerDied","Data":"375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad"} Oct 30 18:22:49.977835 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:49.977752 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="375ea980a79d377be524a6f3ef3eb4b2fac8b13dbf3f74080546dc5b837321ad" Oct 30 18:22:50.647364 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:50.647331 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:22:54.532804 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:22:54.532769 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:22:59.342579 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:59.342540 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz"] Oct 30 18:22:59.353508 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:22:59.353478 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-67382b-7c7sz"] Oct 30 18:23:00.372392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:00.372360 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde14f02-d51c-405a-96b8-05a868bcc9e4" path="/var/lib/kubelet/pods/dde14f02-d51c-405a-96b8-05a868bcc9e4/volumes" Oct 30 18:23:05.667493 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.667455 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerName="main" containerID="cri-o://ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6" gracePeriod=45 Oct 30 18:23:05.795825 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.795785 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.208:8000/ready\": dial tcp 10.134.0.208:8000: connect: connection refused" Oct 30 18:23:05.913532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.913505 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:23:05.961418 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961346 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961418 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961406 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961480 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961499 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961538 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\" (UniqueName: \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961596 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\" (UniqueName: \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\") pod \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\" (UID: \"af1da3e1-9470-4fde-a508-3bf86bf1c38a\") " Oct 30 18:23:05.961643 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961601 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:05.961933 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961742 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:05.961933 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.961812 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:05.962059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.962002 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:05.962059 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.962023 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:05.962171 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.962084 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:05.963940 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.963917 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:05.964169 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.964150 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:05.967403 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:05.967376 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4" pod "af1da3e1-9470-4fde-a508-3bf86bf1c38a" (UID: "af1da3e1-9470-4fde-a508-3bf86bf1c38a"). InnerVolumeSpecName "kube-api-access-dbqf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:06.061461 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.061424 4622 generic.go:356] "Generic (PLEG): container finished" podID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerID="ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6" exitCode=0 Oct 30 18:23:06.061629 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.061469 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" event={"ID":"af1da3e1-9470-4fde-a508-3bf86bf1c38a","Type":"ContainerDied","Data":"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6"} Oct 30 18:23:06.061629 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.061491 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" Oct 30 18:23:06.061629 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.061507 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c" event={"ID":"af1da3e1-9470-4fde-a508-3bf86bf1c38a","Type":"ContainerDied","Data":"c04e2d00632ecc6111e6ea5b5f15ed459e0c79c408a795f066982174e2818b34"} Oct 30 18:23:06.061629 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.061526 4622 scope.go:122] "RemoveContainer" containerID="ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6" Oct 30 18:23:06.062592 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.062569 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af1da3e1-9470-4fde-a508-3bf86bf1c38a-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.062592 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.062594 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbqf4\" (UniqueName: \"kubernetes.io/projected/af1da3e1-9470-4fde-a508-3bf86bf1c38a-kube-api-access-dbqf4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.062741 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.062607 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af1da3e1-9470-4fde-a508-3bf86bf1c38a-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:23:06.082773 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.082744 4622 scope.go:122] "RemoveContainer" containerID="ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6" Oct 30 18:23:06.083144 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:23:06.083122 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6\": container with ID starting with ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6 not found: ID does not exist" containerID="ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6" Oct 30 18:23:06.083251 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.083157 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6"} err="failed to get container status \"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6\": rpc error: code = NotFound desc = could not find container \"ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6\": container with ID starting with ff82776251780deded2fccbdd3c750bd15eb339a5e21a8b9d0db4d50adfecfe6 not found: ID does not exist" Oct 30 18:23:06.115587 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.115552 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:23:06.121839 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.121804 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-67382b/simulator-single-main-67382b-kserve-78654646d6-pgr5c"] Oct 30 18:23:06.372975 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:06.372941 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" path="/var/lib/kubelet/pods/af1da3e1-9470-4fde-a508-3bf86bf1c38a/volumes" Oct 30 18:23:21.251645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.251604 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:23:21.252307 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252281 4622 cpu_manager.go:414] "containerMap: removing container" podUID="dde14f02-d51c-405a-96b8-05a868bcc9e4" containerName="revoke-keys" Oct 30 18:23:21.252307 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252306 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde14f02-d51c-405a-96b8-05a868bcc9e4" containerName="revoke-keys" Oct 30 18:23:21.252480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252378 4622 cpu_manager.go:414] "containerMap: removing container" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerName="main" Oct 30 18:23:21.252480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252388 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerName="main" Oct 30 18:23:21.252554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252482 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1da3e1-9470-4fde-a508-3bf86bf1c38a" containerName="main" Oct 30 18:23:21.252554 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.252494 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde14f02-d51c-405a-96b8-05a868bcc9e4" containerName="revoke-keys" Oct 30 18:23:21.255863 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.255844 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.274846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.274815 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:23:21.282381 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.282355 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:23:21.282574 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.282389 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:23:21.282574 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.282439 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-47a62668-kserve-self-signed-certs\"" Oct 30 18:23:21.282722 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.282495 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hbtff\"" Oct 30 18:23:21.398009 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.397977 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.398009 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.398013 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltfq\" (UniqueName: \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.398308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.398036 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.398308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.398159 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.398308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.398180 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.398308 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.398211 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499519 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499563 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499567 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltfq\" (UniqueName: \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499590 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499650 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499666 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.499841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.499698 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.500079 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.500051 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.500134 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.500085 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.500174 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.500139 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.501868 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.501809 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.502097 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.502079 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.520837 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.520812 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltfq\" (UniqueName: \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\") pod \"e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.568756 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.568721 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:21.865455 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:21.865424 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:23:21.868323 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:23:21.868295 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6337a0bd_cb28_4ca4_8867_c3b86553e123.slice/crio-a7102381b81236e7d9e381ccacdff77aa7bd6fd63a6c334fa9e81e3d52d75361 WatchSource:0}: Error finding container a7102381b81236e7d9e381ccacdff77aa7bd6fd63a6c334fa9e81e3d52d75361: Status 404 returned error can't find the container with id a7102381b81236e7d9e381ccacdff77aa7bd6fd63a6c334fa9e81e3d52d75361 Oct 30 18:23:22.145072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:22.144982 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" event={"ID":"6337a0bd-cb28-4ca4-8867-c3b86553e123","Type":"ContainerStarted","Data":"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304"} Oct 30 18:23:22.145072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:22.145028 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" event={"ID":"6337a0bd-cb28-4ca4-8867-c3b86553e123","Type":"ContainerStarted","Data":"a7102381b81236e7d9e381ccacdff77aa7bd6fd63a6c334fa9e81e3d52d75361"} Oct 30 18:23:22.183055 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:22.182993 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" podStartSLOduration=1.182977194 podStartE2EDuration="1.182977194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:22.18193325 +0000 UTC m=+4229.959405217" watchObservedRunningTime="2026-10-30 18:23:22.182977194 +0000 UTC m=+4229.960449151" Oct 30 18:23:22.569329 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:22.569299 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:24.536007 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:23:24.535963 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:23:29.027798 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:29.027762 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb"] Oct 30 18:23:29.037635 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:29.037603 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31c750cc-mnmvb"] Oct 30 18:23:30.372942 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:30.372910 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c048e62-fc80-43d3-bb85-f056e4ac527f" path="/var/lib/kubelet/pods/0c048e62-fc80-43d3-bb85-f056e4ac527f/volumes" Oct 30 18:23:31.569323 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:31.569292 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:31.581562 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:31.581542 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:32.256397 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:32.256365 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:23:42.396625 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.396586 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:23:42.400702 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.400680 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.424791 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.424764 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-47a62668-kserve-self-signed-certs\"" Oct 30 18:23:42.427450 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.427421 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:23:42.446271 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446233 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.446386 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446370 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.446433 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446396 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.446433 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446416 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfn44\" (UniqueName: \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.446515 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446488 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.446560 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.446523 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547395 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547365 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547400 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547428 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfn44\" (UniqueName: \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547466 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547497 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547545 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547882 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547814 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.547946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547910 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.548007 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.547963 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.549666 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.549645 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.550061 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.550045 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.568731 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.568705 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfn44\" (UniqueName: \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\") pod \"e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:42.715034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:42.714951 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:43.009402 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:43.009375 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:23:43.011311 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:23:43.011280 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5dc9543_8988_43b6_8871_2d5cac6f5d7e.slice/crio-db2e9183567b92f5274a638aa74b3b4f788f4264ae7a73a0baa0471c0950ab1c WatchSource:0}: Error finding container db2e9183567b92f5274a638aa74b3b4f788f4264ae7a73a0baa0471c0950ab1c: Status 404 returned error can't find the container with id db2e9183567b92f5274a638aa74b3b4f788f4264ae7a73a0baa0471c0950ab1c Oct 30 18:23:43.307322 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:43.307287 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" event={"ID":"f5dc9543-8988-43b6-8871-2d5cac6f5d7e","Type":"ContainerStarted","Data":"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1"} Oct 30 18:23:43.307505 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:43.307340 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" event={"ID":"f5dc9543-8988-43b6-8871-2d5cac6f5d7e","Type":"ContainerStarted","Data":"db2e9183567b92f5274a638aa74b3b4f788f4264ae7a73a0baa0471c0950ab1c"} Oct 30 18:23:43.346698 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:43.346644 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" podStartSLOduration=1.346627947 podStartE2EDuration="1.346627947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:43.345125919 +0000 UTC m=+4251.122597875" watchObservedRunningTime="2026-10-30 18:23:43.346627947 +0000 UTC m=+4251.124099905" Oct 30 18:23:43.715989 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:43.715903 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:52.715764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:52.715684 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:52.728006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:52.727984 4622 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:53.370463 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:23:53.370429 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:23:54.532529 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:23:54.532492 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:24:21.054071 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:21.054030 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:24:23.974672 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:23.974601 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:24:24.561285 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:24:24.558398 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:24:30.029842 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:30.029794 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4l8zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:30.030273 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:30.029870 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:30.069645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:30.069352 4622 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-2kfst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:30.069645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:30.069420 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:36.075707 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.075645 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerName="main" containerID="cri-o://0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304" gracePeriod=45 Oct 30 18:24:36.320637 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.320612 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:24:36.478394 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478328 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478394 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478375 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\" (UniqueName: \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478397 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478439 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\" (UniqueName: \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478458 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478575 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478506 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\") pod \"6337a0bd-cb28-4ca4-8867-c3b86553e123\" (UID: \"6337a0bd-cb28-4ca4-8867-c3b86553e123\") " Oct 30 18:24:36.478800 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478695 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:36.478869 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478808 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:36.478869 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478810 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.478869 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.478834 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:36.480693 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.480670 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:36.480693 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.480666 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:36.480886 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.480672 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq" pod "6337a0bd-cb28-4ca4-8867-c3b86553e123" (UID: "6337a0bd-cb28-4ca4-8867-c3b86553e123"). InnerVolumeSpecName "kube-api-access-jltfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:36.579956 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.579935 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.579956 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.579955 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.580090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.579964 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltfq\" (UniqueName: \"kubernetes.io/projected/6337a0bd-cb28-4ca4-8867-c3b86553e123-kube-api-access-jltfq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.580090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.579973 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6337a0bd-cb28-4ca4-8867-c3b86553e123-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.580090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.579981 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6337a0bd-cb28-4ca4-8867-c3b86553e123-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:36.664903 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.664882 4622 generic.go:356] "Generic (PLEG): container finished" podID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerID="0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304" exitCode=0 Oct 30 18:24:36.665008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.664925 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" Oct 30 18:24:36.665008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.664966 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" event={"ID":"6337a0bd-cb28-4ca4-8867-c3b86553e123","Type":"ContainerDied","Data":"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304"} Oct 30 18:24:36.665008 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.665003 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" event={"ID":"6337a0bd-cb28-4ca4-8867-c3b86553e123","Type":"ContainerDied","Data":"a7102381b81236e7d9e381ccacdff77aa7bd6fd63a6c334fa9e81e3d52d75361"} Oct 30 18:24:36.665122 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.665019 4622 scope.go:122] "RemoveContainer" containerID="0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304" Oct 30 18:24:36.674619 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.674603 4622 scope.go:122] "RemoveContainer" containerID="0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304" Oct 30 18:24:36.674905 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:24:36.674884 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304\": container with ID starting with 0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304 not found: ID does not exist" containerID="0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304" Oct 30 18:24:36.674979 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.674914 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304"} err="failed to get container status \"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304\": rpc error: code = NotFound desc = could not find container \"0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304\": container with ID starting with 0ef29d9096f17a388e1bd735dccc351598ed5a0937bbbc474b40278e01f79304 not found: ID does not exist" Oct 30 18:24:36.711018 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.710993 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:24:36.714591 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:36.714567 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl"] Oct 30 18:24:37.246367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:37.246330 4622 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-47a62668-kserve-766fb4cf6f-2l5dl" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerName="main" probeResult="failure" output="Get \"https://10.134.0.211:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.372633 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:38.372600 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" path="/var/lib/kubelet/pods/6337a0bd-cb28-4ca4-8867-c3b86553e123/volumes" Oct 30 18:24:39.029721 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.029685 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" podUID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" containerName="main" containerID="cri-o://d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1" gracePeriod=45 Oct 30 18:24:39.276748 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.276722 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403546 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403631 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403673 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403765 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403794 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.404291 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.403883 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\" (UniqueName: \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\") pod \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\" (UID: \"f5dc9543-8988-43b6-8871-2d5cac6f5d7e\") " Oct 30 18:24:39.407299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.406744 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:39.407299 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.407244 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:39.414364 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.414335 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:39.437828 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.437721 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "kube-api-access-mfn44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:39.437828 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.437783 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:39.439392 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.439354 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm" pod "f5dc9543-8988-43b6-8871-2d5cac6f5d7e" (UID: "f5dc9543-8988-43b6-8871-2d5cac6f5d7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506874 4622 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tls-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506910 4622 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-home\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506924 4622 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-tmp-dir\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506936 4622 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-dshm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506949 4622 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-model-cache\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.507350 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.506962 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfn44\" (UniqueName: \"kubernetes.io/projected/f5dc9543-8988-43b6-8871-2d5cac6f5d7e-kube-api-access-mfn44\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:24:39.711530 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.711432 4622 generic.go:356] "Generic (PLEG): container finished" podID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" containerID="d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1" exitCode=0 Oct 30 18:24:39.711712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.711579 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" event={"ID":"f5dc9543-8988-43b6-8871-2d5cac6f5d7e","Type":"ContainerDied","Data":"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1"} Oct 30 18:24:39.711712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.711614 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" event={"ID":"f5dc9543-8988-43b6-8871-2d5cac6f5d7e","Type":"ContainerDied","Data":"db2e9183567b92f5274a638aa74b3b4f788f4264ae7a73a0baa0471c0950ab1c"} Oct 30 18:24:39.711712 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.711637 4622 scope.go:122] "RemoveContainer" containerID="d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1" Oct 30 18:24:39.711902 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.711764 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49" Oct 30 18:24:39.727029 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.727009 4622 scope.go:122] "RemoveContainer" containerID="d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1" Oct 30 18:24:39.727300 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:24:39.727275 4622 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1\": container with ID starting with d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1 not found: ID does not exist" containerID="d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1" Oct 30 18:24:39.727369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.727311 4622 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1"} err="failed to get container status \"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1\": rpc error: code = NotFound desc = could not find container \"d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1\": container with ID starting with d6ae7b0bbcefa39fe75a9dabceec71df282f16a618de2c4c0bb3723d8926f0b1 not found: ID does not exist" Oct 30 18:24:39.779633 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.779572 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:24:39.791290 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:39.783229 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-47a62668-kserve-57d5dbffd5-srm49"] Oct 30 18:24:40.399305 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:40.393502 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" path="/var/lib/kubelet/pods/f5dc9543-8988-43b6-8871-2d5cac6f5d7e/volumes" Oct 30 18:24:54.537356 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:24:54.537319 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:24:55.374089 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:55.374061 4622 scope.go:122] "RemoveContainer" containerID="9923f8d78c81fb8fe194caf51bdec9e122b35735f8722068da872d20fabe5045" Oct 30 18:24:55.385034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:24:55.385011 4622 scope.go:122] "RemoveContainer" containerID="1db0c388715df32088b84fde1e223d04f662ea9f3bc7070a9a8ddfc14928e402" Oct 30 18:25:24.532472 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:25:24.532391 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:25:54.532635 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:25:54.532596 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:25:54.561462 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.561438 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s"] Oct 30 18:25:54.561860 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.561843 4622 cpu_manager.go:414] "containerMap: removing container" podUID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" containerName="main" Oct 30 18:25:54.561950 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.561862 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" containerName="main" Oct 30 18:25:54.561950 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.561896 4622 cpu_manager.go:414] "containerMap: removing container" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerName="main" Oct 30 18:25:54.561950 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.561905 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerName="main" Oct 30 18:25:54.562113 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.562014 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dc9543-8988-43b6-8871-2d5cac6f5d7e" containerName="main" Oct 30 18:25:54.562113 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.562031 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="6337a0bd-cb28-4ca4-8867-c3b86553e123" containerName="main" Oct 30 18:25:54.564914 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.564893 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.588989 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.588958 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:25:54.588989 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.588966 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:54.589184 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.588965 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:25:54.589184 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.588973 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:25:54.592576 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.592550 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s"] Oct 30 18:25:54.650658 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.650623 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.650782 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.650745 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzml\" (UniqueName: \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.650846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.650785 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.752021 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.751998 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzml\" (UniqueName: \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.752159 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.752033 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.752159 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.752106 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.752799 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.752770 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.754249 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.754232 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.773085 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.773065 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzml\" (UniqueName: \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\") pod \"maas-api-revoke-sub-f03629db-lhq2s\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:54.877373 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:54.877302 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:55.174348 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:55.174325 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s"] Oct 30 18:25:55.177131 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:25:55.177098 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c31ac1_6bce_4bd1_a91b_9cf7cc6e6161.slice/crio-c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854 WatchSource:0}: Error finding container c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854: Status 404 returned error can't find the container with id c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854 Oct 30 18:25:56.158765 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:56.158734 4622 generic.go:356] "Generic (PLEG): container finished" podID="b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" containerID="033e430f359d4934f686d851ba5ec5cbfb9b8c8e87d5d89497013ef642dec3ba" exitCode=0 Oct 30 18:25:56.159092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:56.158781 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" event={"ID":"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161","Type":"ContainerDied","Data":"033e430f359d4934f686d851ba5ec5cbfb9b8c8e87d5d89497013ef642dec3ba"} Oct 30 18:25:56.159092 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:56.158841 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" event={"ID":"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161","Type":"ContainerStarted","Data":"c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854"} Oct 30 18:25:57.287491 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.287467 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:57.372926 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.372894 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\" (UniqueName: \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\") pod \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " Oct 30 18:25:57.373072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.372939 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\") pod \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " Oct 30 18:25:57.373072 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.373031 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\") pod \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\" (UID: \"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161\") " Oct 30 18:25:57.373382 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.373341 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca" pod "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" (UID: "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:57.374870 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.374848 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml" pod "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" (UID: "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161"). InnerVolumeSpecName "kube-api-access-2gzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:57.374967 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.374947 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca" pod "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" (UID: "b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:57.474697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.474639 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzml\" (UniqueName: \"kubernetes.io/projected/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-kube-api-access-2gzml\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:25:57.474697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.474662 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:25:57.474697 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:57.474671 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:25:58.170316 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:58.170284 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" Oct 30 18:25:58.170519 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:58.170316 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s" event={"ID":"b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161","Type":"ContainerDied","Data":"c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854"} Oct 30 18:25:58.170519 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:25:58.170391 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8efa18bcf43b53b3e7e5c56200a6b371e42ff14d25c64440b5a87e401fc7854" Oct 30 18:26:12.667146 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.667108 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276"] Oct 30 18:26:12.667548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.667522 4622 cpu_manager.go:414] "containerMap: removing container" podUID="b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" containerName="revoke-keys" Oct 30 18:26:12.667548 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.667535 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" containerName="revoke-keys" Oct 30 18:26:12.667630 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.667619 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" containerName="revoke-keys" Oct 30 18:26:12.670758 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.670738 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.684018 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.683985 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:12.684018 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.683986 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:12.694630 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.694593 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:12.694752 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.694642 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:26:12.697805 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.697778 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276"] Oct 30 18:26:12.844547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.844508 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89lj\" (UniqueName: \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.844547 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.844550 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.844749 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.844670 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.946067 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.945999 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.946189 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.946075 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.946189 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.946105 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89lj\" (UniqueName: \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.946729 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.946706 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.948091 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.948069 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.967793 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.967773 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89lj\" (UniqueName: \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\") pod \"maas-api-revoke-sub-0165d7d2-qq276\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:12.983583 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:12.983561 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:13.274228 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:13.274205 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276"] Oct 30 18:26:13.276659 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:26:13.276630 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dbfd07_a6a1_46ea_920b_006db4c39ddb.slice/crio-ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d WatchSource:0}: Error finding container ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d: Status 404 returned error can't find the container with id ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d Oct 30 18:26:14.259132 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:14.259058 4622 generic.go:356] "Generic (PLEG): container finished" podID="54dbfd07-a6a1-46ea-920b-006db4c39ddb" containerID="bdc2ef5320aaa388c7823c8c9ab8b882ba5156f3259d2a489900c858d859f138" exitCode=0 Oct 30 18:26:14.259498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:14.259143 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" event={"ID":"54dbfd07-a6a1-46ea-920b-006db4c39ddb","Type":"ContainerDied","Data":"bdc2ef5320aaa388c7823c8c9ab8b882ba5156f3259d2a489900c858d859f138"} Oct 30 18:26:14.259498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:14.259199 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" event={"ID":"54dbfd07-a6a1-46ea-920b-006db4c39ddb","Type":"ContainerStarted","Data":"ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d"} Oct 30 18:26:15.393698 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.393674 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:15.567587 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.567562 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\" (UniqueName: \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\") pod \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " Oct 30 18:26:15.567736 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.567598 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\") pod \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " Oct 30 18:26:15.567736 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.567631 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\") pod \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\" (UID: \"54dbfd07-a6a1-46ea-920b-006db4c39ddb\") " Oct 30 18:26:15.568022 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.567999 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca" pod "54dbfd07-a6a1-46ea-920b-006db4c39ddb" (UID: "54dbfd07-a6a1-46ea-920b-006db4c39ddb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:15.569634 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.569609 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca" pod "54dbfd07-a6a1-46ea-920b-006db4c39ddb" (UID: "54dbfd07-a6a1-46ea-920b-006db4c39ddb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:15.569733 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.569659 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj" pod "54dbfd07-a6a1-46ea-920b-006db4c39ddb" (UID: "54dbfd07-a6a1-46ea-920b-006db4c39ddb"). InnerVolumeSpecName "kube-api-access-x89lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:15.668579 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.668555 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89lj\" (UniqueName: \"kubernetes.io/projected/54dbfd07-a6a1-46ea-920b-006db4c39ddb-kube-api-access-x89lj\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.668579 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.668578 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.668724 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:15.668587 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54dbfd07-a6a1-46ea-920b-006db4c39ddb-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:16.270803 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:16.270774 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" Oct 30 18:26:16.270974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:16.270819 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276" event={"ID":"54dbfd07-a6a1-46ea-920b-006db4c39ddb","Type":"ContainerDied","Data":"ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d"} Oct 30 18:26:16.270974 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:16.270866 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1830c99adafd7a6b9c4e8a39f57b0b3f255c2143099b278d5ba82ea281d76d" Oct 30 18:26:23.632242 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.632208 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc"] Oct 30 18:26:23.632759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.632637 4622 cpu_manager.go:414] "containerMap: removing container" podUID="54dbfd07-a6a1-46ea-920b-006db4c39ddb" containerName="revoke-keys" Oct 30 18:26:23.632759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.632652 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54dbfd07-a6a1-46ea-920b-006db4c39ddb" containerName="revoke-keys" Oct 30 18:26:23.632759 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.632725 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="54dbfd07-a6a1-46ea-920b-006db4c39ddb" containerName="revoke-keys" Oct 30 18:26:23.635790 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.635770 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.659175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.659149 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:23.659351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.659177 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:26:23.659351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.659177 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:26:23.659351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.659215 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:26:23.662592 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.662569 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc"] Oct 30 18:26:23.731314 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.731285 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfrm\" (UniqueName: \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.731434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.731329 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.731434 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.731406 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.832859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.832834 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfrm\" (UniqueName: \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.832969 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.832866 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.832969 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.832895 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.833484 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.833464 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.835149 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.835128 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.853846 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.853826 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfrm\" (UniqueName: \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\") pod \"maas-api-revoke-sub-6b19338f-gd4mc\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:23.948033 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:23.947979 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:24.242955 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:24.242927 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc"] Oct 30 18:26:24.244783 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:26:24.244752 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb56209_9a96_42f3_9677_35162ab9ddcd.slice/crio-0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2 WatchSource:0}: Error finding container 0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2: Status 404 returned error can't find the container with id 0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2 Oct 30 18:26:24.312959 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:24.312931 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" event={"ID":"2cb56209-9a96-42f3-9677-35162ab9ddcd","Type":"ContainerStarted","Data":"0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2"} Oct 30 18:26:24.532373 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:26:24.532298 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:26:25.319658 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:25.319624 4622 generic.go:356] "Generic (PLEG): container finished" podID="2cb56209-9a96-42f3-9677-35162ab9ddcd" containerID="9f41503ade861c859b580a0f72d9b2af5f328190c2d95faeb687067b43ae317d" exitCode=0 Oct 30 18:26:25.320031 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:25.319705 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" event={"ID":"2cb56209-9a96-42f3-9677-35162ab9ddcd","Type":"ContainerDied","Data":"9f41503ade861c859b580a0f72d9b2af5f328190c2d95faeb687067b43ae317d"} Oct 30 18:26:26.459821 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.459794 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:26.559027 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.559000 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\") pod \"2cb56209-9a96-42f3-9677-35162ab9ddcd\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " Oct 30 18:26:26.559191 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.559073 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\") pod \"2cb56209-9a96-42f3-9677-35162ab9ddcd\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " Oct 30 18:26:26.559191 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.559123 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\" (UniqueName: \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\") pod \"2cb56209-9a96-42f3-9677-35162ab9ddcd\" (UID: \"2cb56209-9a96-42f3-9677-35162ab9ddcd\") " Oct 30 18:26:26.559485 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.559460 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca" pod "2cb56209-9a96-42f3-9677-35162ab9ddcd" (UID: "2cb56209-9a96-42f3-9677-35162ab9ddcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:26.560997 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.560968 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca" pod "2cb56209-9a96-42f3-9677-35162ab9ddcd" (UID: "2cb56209-9a96-42f3-9677-35162ab9ddcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:26.561104 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.561074 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm" pod "2cb56209-9a96-42f3-9677-35162ab9ddcd" (UID: "2cb56209-9a96-42f3-9677-35162ab9ddcd"). InnerVolumeSpecName "kube-api-access-czfrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:26.660125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.660060 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfrm\" (UniqueName: \"kubernetes.io/projected/2cb56209-9a96-42f3-9677-35162ab9ddcd-kube-api-access-czfrm\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:26.660125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.660085 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-xks-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:26.660125 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:26.660098 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cb56209-9a96-42f3-9677-35162ab9ddcd-maas-api-openshift-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:26:27.334281 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:27.334227 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" Oct 30 18:26:27.334281 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:27.334237 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc" event={"ID":"2cb56209-9a96-42f3-9677-35162ab9ddcd","Type":"ContainerDied","Data":"0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2"} Oct 30 18:26:27.334555 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:27.334319 4622 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ce924614b6e5c011e359ddf2acd088f44ca301e167f2e3b788f16fba1fb84a2" Oct 30 18:26:54.537018 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:26:54.536929 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:26:56.049723 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:56.049692 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8tfhq_3e345ce8-1709-483f-845b-d4f00a9980b9/manager/0.log" Oct 30 18:26:56.528551 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:56.528480 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m9k4s_55c6457c-298e-4c2c-9f30-f6b36738a507/manager/0.log" Oct 30 18:26:57.046146 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:57.046115 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pf6jq_0e04659f-ddde-43e6-96a2-d34c17d0cc25/server/0.log" Oct 30 18:26:57.241781 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:57.241753 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-29dgg_cf94a5f2-82ff-4636-a99c-0a36d60d180b/manager/0.log" Oct 30 18:26:57.463993 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:57.463913 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-tph5f_eee295a7-b23f-4e56-a41a-642cebd563e6/manager/0.log" Oct 30 18:26:57.706192 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:57.706166 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5_eeda50f8-a5eb-49b3-89bd-434056f44cbe/copy-manifests/0.log" Oct 30 18:26:57.737670 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:26:57.737618 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5_eeda50f8-a5eb-49b3-89bd-434056f44cbe/manager/0.log" Oct 30 18:27:19.735083 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:19.735046 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qq276_54dbfd07-a6a1-46ea-920b-006db4c39ddb/revoke-keys/0.log" Oct 30 18:27:20.508663 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:20.508630 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gd4mc_2cb56209-9a96-42f3-9677-35162ab9ddcd/revoke-keys/0.log" Oct 30 18:27:20.717486 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:20.717453 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c334f6dd-nkm2g_4aceb7a9-3572-445b-aeb3-ca6c875698c3/revoke-keys/0.log" Oct 30 18:27:20.920071 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:20.920041 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lhq2s_b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161/revoke-keys/0.log" Oct 30 18:27:23.152121 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:23.152080 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-vbpd7_05be6fab-f98b-42cf-9c25-9b4605b47504/manager/0.log" Oct 30 18:27:24.020370 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:24.020333 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-292f8_f7fa3d37-0764-48d9-8936-3bdbafd810e5/manager/0.log" Oct 30 18:27:24.533015 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:27:24.532979 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:27:24.829972 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:24.829899 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9cpm72l_43d1c03c-734f-42e3-98ac-d7e50594a946/istio-proxy/0.log" Oct 30 18:27:25.587665 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:25.587635 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-2kfst_8bb93d64-c87c-41fb-9ecc-909def18795c/istio-proxy/0.log" Oct 30 18:27:25.795735 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:25.795700 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-4l8zv_4e692f06-e611-4656-8edc-6de03b31b126/istio-proxy/0.log" Oct 30 18:27:36.391119 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:36.391084 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-b7p7t_a7411194-34e7-4d6e-8415-efb9c6e32fc1/istio-proxy/0.log" Oct 30 18:27:38.026608 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:38.026572 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g"] Oct 30 18:27:38.036642 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:38.036618 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c334f6dd-nkm2g"] Oct 30 18:27:38.372000 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:27:38.371969 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aceb7a9-3572-445b-aeb3-ca6c875698c3" path="/var/lib/kubelet/pods/4aceb7a9-3572-445b-aeb3-ca6c875698c3/volumes" Oct 30 18:27:54.532541 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:27:54.532506 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:28:24.533287 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:28:24.533193 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:28:30.493177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:30.493143 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-8tfhq_3e345ce8-1709-483f-845b-d4f00a9980b9/manager/0.log" Oct 30 18:28:30.916498 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:30.916464 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-m9k4s_55c6457c-298e-4c2c-9f30-f6b36738a507/manager/0.log" Oct 30 18:28:41.422801 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:41.422772 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-pf6jq_0e04659f-ddde-43e6-96a2-d34c17d0cc25/server/0.log" Oct 30 18:28:41.624841 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:41.624805 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-29dgg_cf94a5f2-82ff-4636-a99c-0a36d60d180b/manager/0.log" Oct 30 18:28:41.855194 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:41.855161 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-tph5f_eee295a7-b23f-4e56-a41a-642cebd563e6/manager/0.log" Oct 30 18:28:42.104399 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:42.104370 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5_eeda50f8-a5eb-49b3-89bd-434056f44cbe/copy-manifests/0.log" Oct 30 18:28:42.138332 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:42.138250 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c8cd7c7d9-8gvd5_eeda50f8-a5eb-49b3-89bd-434056f44cbe/manager/0.log" Oct 30 18:28:53.827845 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:53.827813 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qq276_54dbfd07-a6a1-46ea-920b-006db4c39ddb/revoke-keys/0.log" Oct 30 18:28:54.532444 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:28:54.532407 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:28:55.438609 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:55.438571 4622 scope.go:122] "RemoveContainer" containerID="11f29f2b323209a59b2606250d55f7ef05f998d4e0a7f17fab0b121c1e3e2474" Oct 30 18:28:55.447588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:28:55.447567 4622 scope.go:122] "RemoveContainer" containerID="6bbbd9e831781328c2809fab1536f7129f6f2825b0251dde536f1862b2888d50" Oct 30 18:29:04.599876 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:04.599849 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-gd4mc_2cb56209-9a96-42f3-9677-35162ab9ddcd/revoke-keys/0.log" Oct 30 18:29:04.803347 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:04.803318 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-lhq2s_b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161/revoke-keys/0.log" Oct 30 18:29:06.621992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:06.621956 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-vbpd7_05be6fab-f98b-42cf-9c25-9b4605b47504/manager/0.log" Oct 30 18:29:07.481680 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:07.481651 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-292f8_f7fa3d37-0764-48d9-8936-3bdbafd810e5/manager/0.log" Oct 30 18:29:18.089280 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:18.089226 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9cpm72l_43d1c03c-734f-42e3-98ac-d7e50594a946/istio-proxy/0.log" Oct 30 18:29:18.963212 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:18.963176 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-2kfst_8bb93d64-c87c-41fb-9ecc-909def18795c/istio-proxy/0.log" Oct 30 18:29:19.188250 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:19.188214 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-4l8zv_4e692f06-e611-4656-8edc-6de03b31b126/istio-proxy/0.log" Oct 30 18:29:19.793369 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:29:19.793337 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-b7p7t_a7411194-34e7-4d6e-8415-efb9c6e32fc1/istio-proxy/0.log" Oct 30 18:29:24.531758 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:29:24.531722 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:29:54.533347 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:29:54.533256 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:30:00.163965 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.163925 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:00.164489 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.164336 4622 cpu_manager.go:414] "containerMap: removing container" podUID="2cb56209-9a96-42f3-9677-35162ab9ddcd" containerName="revoke-keys" Oct 30 18:30:00.164489 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.164349 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb56209-9a96-42f3-9677-35162ab9ddcd" containerName="revoke-keys" Oct 30 18:30:00.164489 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.164444 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb56209-9a96-42f3-9677-35162ab9ddcd" containerName="revoke-keys" Oct 30 18:30:00.167717 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.167698 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.191847 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.191822 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:30:00.191847 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.191836 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:30:00.192006 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.191824 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n59qx\"" Oct 30 18:30:00.203480 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.203450 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:00.269050 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.269015 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrn4\" (UniqueName: \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.269215 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.269075 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.369687 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.369649 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.369851 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.369793 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrn4\" (UniqueName: \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.370322 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.370302 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.392851 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.392825 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrn4\" (UniqueName: \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\") pod \"maas-api-key-cleanup-29889750-h4fx7\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.480335 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.480254 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:00.774231 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:00.774199 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:00.777104 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:30:00.777073 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51c9793_92eb_42c9_b597_4f7836c030b3.slice/crio-40eeffb087c4b8fd31a8bc5264b24fb69b2ec96f329a759f59f26840be341f8a WatchSource:0}: Error finding container 40eeffb087c4b8fd31a8bc5264b24fb69b2ec96f329a759f59f26840be341f8a: Status 404 returned error can't find the container with id 40eeffb087c4b8fd31a8bc5264b24fb69b2ec96f329a759f59f26840be341f8a Oct 30 18:30:01.459606 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:01.459575 4622 generic.go:356] "Generic (PLEG): container finished" podID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerID="e70fdc0960bc69733c31a8d2f765d9d51cbb7dd331d8e3be85bb8433cfcd545c" exitCode=60 Oct 30 18:30:01.459925 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:01.459659 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" event={"ID":"c51c9793-92eb-42c9-b597-4f7836c030b3","Type":"ContainerDied","Data":"e70fdc0960bc69733c31a8d2f765d9d51cbb7dd331d8e3be85bb8433cfcd545c"} Oct 30 18:30:01.459925 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:01.459715 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" event={"ID":"c51c9793-92eb-42c9-b597-4f7836c030b3","Type":"ContainerStarted","Data":"40eeffb087c4b8fd31a8bc5264b24fb69b2ec96f329a759f59f26840be341f8a"} Oct 30 18:30:01.459925 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:01.459853 4622 scope.go:122] "RemoveContainer" containerID="e70fdc0960bc69733c31a8d2f765d9d51cbb7dd331d8e3be85bb8433cfcd545c" Oct 30 18:30:02.466100 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:02.466021 4622 generic.go:356] "Generic (PLEG): container finished" podID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerID="904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691" exitCode=60 Oct 30 18:30:02.466100 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:02.466081 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" event={"ID":"c51c9793-92eb-42c9-b597-4f7836c030b3","Type":"ContainerDied","Data":"904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691"} Oct 30 18:30:02.466646 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:02.466132 4622 scope.go:122] "RemoveContainer" containerID="e70fdc0960bc69733c31a8d2f765d9d51cbb7dd331d8e3be85bb8433cfcd545c" Oct 30 18:30:02.466646 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:02.466348 4622 scope.go:122] "RemoveContainer" containerID="904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691" Oct 30 18:30:02.466646 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:30:02.466586 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-h4fx7_odh-ai-gateway-infra(c51c9793-92eb-42c9-b597-4f7836c030b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" Oct 30 18:30:03.473344 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:03.473314 4622 scope.go:122] "RemoveContainer" containerID="904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691" Oct 30 18:30:03.473717 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:30:03.473509 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-h4fx7_odh-ai-gateway-infra(c51c9793-92eb-42c9-b597-4f7836c030b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" Oct 30 18:30:14.367532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:14.367498 4622 scope.go:122] "RemoveContainer" containerID="904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691" Oct 30 18:30:15.533163 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:15.533128 4622 generic.go:356] "Generic (PLEG): container finished" podID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerID="56cc424ed4fe077dab8d72de560731beaf0bf6cb249a86d69c42bf4af694dabf" exitCode=60 Oct 30 18:30:15.533645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:15.533200 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" event={"ID":"c51c9793-92eb-42c9-b597-4f7836c030b3","Type":"ContainerDied","Data":"56cc424ed4fe077dab8d72de560731beaf0bf6cb249a86d69c42bf4af694dabf"} Oct 30 18:30:15.533645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:15.533283 4622 scope.go:122] "RemoveContainer" containerID="904d00298040704730ff2f1a3c93f167dd296b60676c55dd79b24be7bfe73691" Oct 30 18:30:15.533645 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:15.533539 4622 scope.go:122] "RemoveContainer" containerID="56cc424ed4fe077dab8d72de560731beaf0bf6cb249a86d69c42bf4af694dabf" Oct 30 18:30:15.533766 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:30:15.533742 4622 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-h4fx7_odh-ai-gateway-infra(c51c9793-92eb-42c9-b597-4f7836c030b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" Oct 30 18:30:16.581555 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.581511 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:16.714588 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.714565 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:16.798945 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.798909 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\" (UniqueName: \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\") pod \"c51c9793-92eb-42c9-b597-4f7836c030b3\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " Oct 30 18:30:16.799151 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.798962 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\") pod \"c51c9793-92eb-42c9-b597-4f7836c030b3\" (UID: \"c51c9793-92eb-42c9-b597-4f7836c030b3\") " Oct 30 18:30:16.799411 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.799383 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca" pod "c51c9793-92eb-42c9-b597-4f7836c030b3" (UID: "c51c9793-92eb-42c9-b597-4f7836c030b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.801222 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.801196 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4" pod "c51c9793-92eb-42c9-b597-4f7836c030b3" (UID: "c51c9793-92eb-42c9-b597-4f7836c030b3"). InnerVolumeSpecName "kube-api-access-qfrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.899958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.899876 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrn4\" (UniqueName: \"kubernetes.io/projected/c51c9793-92eb-42c9-b597-4f7836c030b3-kube-api-access-qfrn4\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.899958 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:16.899908 4622 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51c9793-92eb-42c9-b597-4f7836c030b3-maas-api-service-ca\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.547158 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:17.547130 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" Oct 30 18:30:17.547367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:17.547175 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7" event={"ID":"c51c9793-92eb-42c9-b597-4f7836c030b3","Type":"ContainerDied","Data":"40eeffb087c4b8fd31a8bc5264b24fb69b2ec96f329a759f59f26840be341f8a"} Oct 30 18:30:17.547367 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:17.547234 4622 scope.go:122] "RemoveContainer" containerID="56cc424ed4fe077dab8d72de560731beaf0bf6cb249a86d69c42bf4af694dabf" Oct 30 18:30:17.593543 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:17.593510 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:17.603896 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:17.603863 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-h4fx7"] Oct 30 18:30:18.373181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:18.373150 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" path="/var/lib/kubelet/pods/c51c9793-92eb-42c9-b597-4f7836c030b3/volumes" Oct 30 18:30:24.533073 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:30:24.533040 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:30:54.532606 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:30:54.532571 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:30:58.029062 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:58.029029 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s"] Oct 30 18:30:58.039391 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:58.039368 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-lhq2s"] Oct 30 18:30:58.372910 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:30:58.372877 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161" path="/var/lib/kubelet/pods/b1c31ac1-6bce-4bd1-a91b-9cf7cc6e6161/volumes" Oct 30 18:31:16.027105 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:16.027070 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276"] Oct 30 18:31:16.037351 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:16.037328 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qq276"] Oct 30 18:31:16.372945 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:16.372913 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54dbfd07-a6a1-46ea-920b-006db4c39ddb" path="/var/lib/kubelet/pods/54dbfd07-a6a1-46ea-920b-006db4c39ddb/volumes" Oct 30 18:31:24.535529 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:31:24.533476 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:31:27.039331 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:27.039301 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc"] Oct 30 18:31:27.048872 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:27.048844 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-gd4mc"] Oct 30 18:31:28.372496 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:28.372465 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb56209-9a96-42f3-9677-35162ab9ddcd" path="/var/lib/kubelet/pods/2cb56209-9a96-42f3-9677-35162ab9ddcd/volumes" Oct 30 18:31:51.830961 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:51.830925 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:31:51.831394 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:51.831127 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" containerID="cri-o://38a6d4a1ca5aa1510b6aab8bb17a89e7a82ec820f49eb2a5e9aedc7cb721399a" gracePeriod=30 Oct 30 18:31:51.841718 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:51.841692 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:31:51.841952 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:51.841921 4622 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" containerID="cri-o://bd4d433490aa4a478c40ba00fbfff870284d0f1204591ccd51f92c148cc6be7b" gracePeriod=30 Oct 30 18:31:54.533071 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:31:54.533025 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:31:55.491500 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:55.491472 4622 scope.go:122] "RemoveContainer" containerID="033e430f359d4934f686d851ba5ec5cbfb9b8c8e87d5d89497013ef642dec3ba" Oct 30 18:31:57.071393 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.071355 4622 generic.go:356] "Generic (PLEG): container finished" podID="4e692f06-e611-4656-8edc-6de03b31b126" containerID="38a6d4a1ca5aa1510b6aab8bb17a89e7a82ec820f49eb2a5e9aedc7cb721399a" exitCode=0 Oct 30 18:31:57.071393 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.071382 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" event={"ID":"4e692f06-e611-4656-8edc-6de03b31b126","Type":"ContainerDied","Data":"38a6d4a1ca5aa1510b6aab8bb17a89e7a82ec820f49eb2a5e9aedc7cb721399a"} Oct 30 18:31:57.073187 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.073154 4622 generic.go:356] "Generic (PLEG): container finished" podID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerID="bd4d433490aa4a478c40ba00fbfff870284d0f1204591ccd51f92c148cc6be7b" exitCode=0 Oct 30 18:31:57.073339 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.073207 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" event={"ID":"8bb93d64-c87c-41fb-9ecc-909def18795c","Type":"ContainerDied","Data":"bd4d433490aa4a478c40ba00fbfff870284d0f1204591ccd51f92c148cc6be7b"} Oct 30 18:31:57.107849 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.107827 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:31:57.110641 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.110623 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:31:57.224784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224701 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.224784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224745 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.224784 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224773 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224801 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224842 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224918 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.224965 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225003 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225041 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225080 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225068 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225104 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225120 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225128 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225141 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225158 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225209 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225253 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225315 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225333 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225368 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\") pod \"4e692f06-e611-4656-8edc-6de03b31b126\" (UID: \"4e692f06-e611-4656-8edc-6de03b31b126\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225414 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225437 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.225487 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225479 4622 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\") pod \"8bb93d64-c87c-41fb-9ecc-909def18795c\" (UID: \"8bb93d64-c87c-41fb-9ecc-909def18795c\") " Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225529 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225605 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225798 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225825 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225906 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225924 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225938 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225951 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.225997 4622 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-data\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.226011 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb93d64-c87c-41fb-9ecc-909def18795c-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.226023 4622 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-workload-certs\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.226037 4622 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-workload-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.226181 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.226090 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.227725 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.227700 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.228003 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.227977 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "kube-api-access-4cflq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.228127 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.228015 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "kube-api-access-rj9zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.228239 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.228212 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.228370 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.228314 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:57.228473 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.228456 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:57.229175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.229156 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token" pod "8bb93d64-c87c-41fb-9ecc-909def18795c" (UID: "8bb93d64-c87c-41fb-9ecc-909def18795c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:57.229175 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.229158 4622 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo" pod "4e692f06-e611-4656-8edc-6de03b31b126" (UID: "4e692f06-e611-4656-8edc-6de03b31b126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:57.326948 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326927 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.326948 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326949 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9zk\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-kube-api-access-rj9zk\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326960 4622 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-istio-envoy\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326969 4622 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cflq\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-kube-api-access-4cflq\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326978 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e692f06-e611-4656-8edc-6de03b31b126-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326986 4622 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e692f06-e611-4656-8edc-6de03b31b126-istiod-ca-cert\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.326994 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e692f06-e611-4656-8edc-6de03b31b126-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.327002 4622 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-podinfo\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.327009 4622 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb93d64-c87c-41fb-9ecc-909def18795c-istio-token\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:57.327096 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:57.327019 4622 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e692f06-e611-4656-8edc-6de03b31b126-credential-socket\") on node \"ip-10-0-143-147.ec2.internal\" DevicePath \"\"" Oct 30 18:31:58.079322 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.079290 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" Oct 30 18:31:58.079773 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.079284 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv" event={"ID":"4e692f06-e611-4656-8edc-6de03b31b126","Type":"ContainerDied","Data":"030b157e7551c419497f566153aed72c8e72d6fd4fce78ce9cfad5212be412eb"} Oct 30 18:31:58.079773 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.079425 4622 scope.go:122] "RemoveContainer" containerID="38a6d4a1ca5aa1510b6aab8bb17a89e7a82ec820f49eb2a5e9aedc7cb721399a" Oct 30 18:31:58.080831 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.080809 4622 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" Oct 30 18:31:58.080909 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.080837 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst" event={"ID":"8bb93d64-c87c-41fb-9ecc-909def18795c","Type":"ContainerDied","Data":"dfab09257f6450f1c12391fa6c7093340b5d9fb3689e667dbbd86ba73ddab9b9"} Oct 30 18:31:58.089869 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.089848 4622 scope.go:122] "RemoveContainer" containerID="bd4d433490aa4a478c40ba00fbfff870284d0f1204591ccd51f92c148cc6be7b" Oct 30 18:31:58.138448 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.138399 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:31:58.148992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.148971 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4l8zv"] Oct 30 18:31:58.176034 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.176015 4622 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:31:58.179749 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.179730 4622 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-2kfst"] Oct 30 18:31:58.372310 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.372279 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e692f06-e611-4656-8edc-6de03b31b126" path="/var/lib/kubelet/pods/4e692f06-e611-4656-8edc-6de03b31b126/volumes" Oct 30 18:31:58.372761 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:31:58.372741 4622 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" path="/var/lib/kubelet/pods/8bb93d64-c87c-41fb-9ecc-909def18795c/volumes" Oct 30 18:32:24.533047 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:32:24.532999 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:32:50.784210 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784171 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjt4j/must-gather-4xj68"] Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784581 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784594 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784609 4622 cpu_manager.go:414] "containerMap: removing container" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784614 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784627 4622 cpu_manager.go:414] "containerMap: removing container" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784632 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784642 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784647 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784654 4622 cpu_manager.go:414] "containerMap: removing container" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784660 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784658 4622 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784711 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784723 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.784992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784730 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb93d64-c87c-41fb-9ecc-909def18795c" containerName="istio-proxy" Oct 30 18:32:50.784992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784735 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e692f06-e611-4656-8edc-6de03b31b126" containerName="istio-proxy" Oct 30 18:32:50.784992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.784843 4622 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51c9793-92eb-42c9-b597-4f7836c030b3" containerName="cleanup" Oct 30 18:32:50.787871 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.787852 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:50.801818 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.801799 4622 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cjt4j\"/\"default-dockercfg-pcvs7\"" Oct 30 18:32:50.812667 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.812640 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"openshift-service-ca.crt\"" Oct 30 18:32:50.812840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.812706 4622 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjt4j\"/\"kube-root-ca.crt\"" Oct 30 18:32:50.814621 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.814600 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/must-gather-4xj68"] Oct 30 18:32:50.955340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.955280 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abcb215c-8544-45b6-a858-2f516b3bb963-must-gather-output\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:50.955340 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:50.955337 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95sls\" (UniqueName: \"kubernetes.io/projected/abcb215c-8544-45b6-a858-2f516b3bb963-kube-api-access-95sls\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.056572 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.056514 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abcb215c-8544-45b6-a858-2f516b3bb963-must-gather-output\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.056676 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.056573 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95sls\" (UniqueName: \"kubernetes.io/projected/abcb215c-8544-45b6-a858-2f516b3bb963-kube-api-access-95sls\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.056836 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.056817 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/abcb215c-8544-45b6-a858-2f516b3bb963-must-gather-output\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.078038 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.078013 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95sls\" (UniqueName: \"kubernetes.io/projected/abcb215c-8544-45b6-a858-2f516b3bb963-kube-api-access-95sls\") pod \"must-gather-4xj68\" (UID: \"abcb215c-8544-45b6-a858-2f516b3bb963\") " pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.101946 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.101925 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/must-gather-4xj68" Oct 30 18:32:51.392771 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.392698 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/must-gather-4xj68"] Oct 30 18:32:51.395807 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:32:51.395777 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabcb215c_8544_45b6_a858_2f516b3bb963.slice/crio-4dbb3849790b3c682387fb839f3c8abdde250bb0a986b3618fa2bebfb9e9d118 WatchSource:0}: Error finding container 4dbb3849790b3c682387fb839f3c8abdde250bb0a986b3618fa2bebfb9e9d118: Status 404 returned error can't find the container with id 4dbb3849790b3c682387fb839f3c8abdde250bb0a986b3618fa2bebfb9e9d118 Oct 30 18:32:51.397549 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:51.397529 4622 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:32:52.356420 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:52.356393 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/must-gather-4xj68" event={"ID":"abcb215c-8544-45b6-a858-2f516b3bb963","Type":"ContainerStarted","Data":"0dd24cf6d259ef83597f2d1bf0643a39be212953f59fd71cc63ee4eb12b918d8"} Oct 30 18:32:52.356744 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:52.356432 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/must-gather-4xj68" event={"ID":"abcb215c-8544-45b6-a858-2f516b3bb963","Type":"ContainerStarted","Data":"4dbb3849790b3c682387fb839f3c8abdde250bb0a986b3618fa2bebfb9e9d118"} Oct 30 18:32:53.364906 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:53.364863 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/must-gather-4xj68" event={"ID":"abcb215c-8544-45b6-a858-2f516b3bb963","Type":"ContainerStarted","Data":"d8d90694ced65e4c2bb3562801544d90e5dea9365961434b28fe5b99083d3f99"} Oct 30 18:32:53.402090 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:53.402044 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cjt4j/must-gather-4xj68" podStartSLOduration=2.678719585 podStartE2EDuration="3.402032457s" totalImagesPullingTime="723.312872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:53.401610749 +0000 UTC m=+4801.179082704" watchObservedRunningTime="2026-10-30 18:32:53.402032457 +0000 UTC m=+4801.179504412" Oct 30 18:32:54.539285 ip-10-0-143-147 kubenswrapper[4622]: E1030 18:32:54.539147 4622 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tzxkz" containerName="extractor" Oct 30 18:32:55.353248 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:55.353217 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-854ml_39f1686a-04e8-4f10-874a-51e855eea9a0/global-pull-secret-syncer/0.log" Oct 30 18:32:55.512992 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:55.512956 4622 scope.go:122] "RemoveContainer" containerID="9f41503ade861c859b580a0f72d9b2af5f328190c2d95faeb687067b43ae317d" Oct 30 18:32:55.522405 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:55.522307 4622 scope.go:122] "RemoveContainer" containerID="bdc2ef5320aaa388c7823c8c9ab8b882ba5156f3259d2a489900c858d859f138" Oct 30 18:32:56.495951 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:56.493552 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rzvbg_aadcc9ba-2535-48cd-abef-1535e15f6ec6/konnectivity-agent/0.log" Oct 30 18:32:56.862334 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:32:56.860703 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-147.ec2.internal_a33e219052cd1905654f6e2cbcaae521/haproxy/0.log" Oct 30 18:33:03.382720 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:03.382683 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-vbpd7_05be6fab-f98b-42cf-9c25-9b4605b47504/manager/0.log" Oct 30 18:33:04.040859 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.035773 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-292f8_f7fa3d37-0764-48d9-8936-3bdbafd810e5/manager/0.log" Oct 30 18:33:04.703201 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.703147 4622 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j"] Oct 30 18:33:04.707014 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.706992 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.722617 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.720983 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j"] Oct 30 18:33:04.804881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.804834 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksx9\" (UniqueName: \"kubernetes.io/projected/3a5387fe-5146-4b28-a524-b1dc93ed04d3-kube-api-access-fksx9\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.804881 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.804888 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-proc\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.805152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.804982 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-lib-modules\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.805152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.805011 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-podres\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.805152 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.805037 4622 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-sys\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906399 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906360 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-lib-modules\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906399 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906403 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-podres\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906421 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-sys\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906454 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksx9\" (UniqueName: \"kubernetes.io/projected/3a5387fe-5146-4b28-a524-b1dc93ed04d3-kube-api-access-fksx9\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906472 4622 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-proc\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906583 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-lib-modules\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906587 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-proc\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906660 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-podres\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.906688 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.906677 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a5387fe-5146-4b28-a524-b1dc93ed04d3-sys\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:04.936177 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:04.936141 4622 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksx9\" (UniqueName: \"kubernetes.io/projected/3a5387fe-5146-4b28-a524-b1dc93ed04d3-kube-api-access-fksx9\") pod \"perf-node-gather-daemonset-gln6j\" (UID: \"3a5387fe-5146-4b28-a524-b1dc93ed04d3\") " pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:05.026917 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.026829 4622 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:05.359840 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.359783 4622 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j"] Oct 30 18:33:05.365485 ip-10-0-143-147 kubenswrapper[4622]: W1030 18:33:05.365456 4622 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a5387fe_5146_4b28_a524_b1dc93ed04d3.slice/crio-951436307e5eab8621f194c115d2700bf622c9ea47c719ddd28aaba8eafc6e99 WatchSource:0}: Error finding container 951436307e5eab8621f194c115d2700bf622c9ea47c719ddd28aaba8eafc6e99: Status 404 returned error can't find the container with id 951436307e5eab8621f194c115d2700bf622c9ea47c719ddd28aaba8eafc6e99 Oct 30 18:33:05.474174 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.474150 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/alertmanager/0.log" Oct 30 18:33:05.523093 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.523053 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" event={"ID":"3a5387fe-5146-4b28-a524-b1dc93ed04d3","Type":"ContainerStarted","Data":"cba8377a1c6942a7a917e2966d999ea7449e78ccafe997aa046e82671b474be8"} Oct 30 18:33:05.523297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.523110 4622 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" event={"ID":"3a5387fe-5146-4b28-a524-b1dc93ed04d3","Type":"ContainerStarted","Data":"951436307e5eab8621f194c115d2700bf622c9ea47c719ddd28aaba8eafc6e99"} Oct 30 18:33:05.523297 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.523132 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:05.559739 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.559682 4622 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" podStartSLOduration=1.5596690629999999 podStartE2EDuration="1.559669063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:05.5592908 +0000 UTC m=+4813.336762752" watchObservedRunningTime="2026-10-30 18:33:05.559669063 +0000 UTC m=+4813.337141037" Oct 30 18:33:05.601628 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.601599 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/config-reloader/0.log" Oct 30 18:33:05.729727 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.729687 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/kube-rbac-proxy-web/0.log" Oct 30 18:33:05.857413 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.857384 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/kube-rbac-proxy/0.log" Oct 30 18:33:05.995659 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:05.995588 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/kube-rbac-proxy-metric/0.log" Oct 30 18:33:06.127471 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:06.126504 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/prom-label-proxy/0.log" Oct 30 18:33:06.253594 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:06.253522 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ef900d92-5256-4606-8198-ee3bfb001552/init-config-reloader/0.log" Oct 30 18:33:06.917390 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:06.917361 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f796dd7cf-fppn4_2347c334-c94f-45a8-b1c3-e15d654d1862/metrics-server/0.log" Oct 30 18:33:07.542411 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:07.542381 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtwmg_500e754b-8401-4604-8d32-5105317e9dc6/node-exporter/0.log" Oct 30 18:33:07.670961 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:07.670934 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtwmg_500e754b-8401-4604-8d32-5105317e9dc6/kube-rbac-proxy/0.log" Oct 30 18:33:07.797042 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:07.796975 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gtwmg_500e754b-8401-4604-8d32-5105317e9dc6/init-textfile/0.log" Oct 30 18:33:09.618764 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:09.618734 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-svgft_f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/prometheus-operator/0.log" Oct 30 18:33:09.737838 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:09.737812 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-svgft_f7f87ebf-74d3-4c17-aa6a-c8dff6f9e08b/kube-rbac-proxy/0.log" Oct 30 18:33:09.867532 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:09.867477 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bk4wv_909cd2de-faa4-4b8f-a0b1-5bc4e261b32c/prometheus-operator-admission-webhook/0.log" Oct 30 18:33:10.387069 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:10.387037 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/thanos-query/0.log" Oct 30 18:33:10.515396 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:10.515356 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/kube-rbac-proxy-web/0.log" Oct 30 18:33:10.647384 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:10.647314 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/kube-rbac-proxy/0.log" Oct 30 18:33:10.773995 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:10.773962 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/prom-label-proxy/0.log" Oct 30 18:33:10.899980 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:10.899891 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/kube-rbac-proxy-rules/0.log" Oct 30 18:33:11.025741 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:11.025699 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78dc685f45-bspxf_c735f137-8697-422b-b3e6-95852c822b4e/kube-rbac-proxy-metrics/0.log" Oct 30 18:33:11.538148 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:11.538120 4622 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cjt4j/perf-node-gather-daemonset-gln6j" Oct 30 18:33:14.708349 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:14.708313 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-774c8b44ff-w4tz5_d24edaf4-dbf7-4fbb-96e6-0f6275e04572/console/0.log" Oct 30 18:33:17.414472 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:17.414425 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9cpm72l_43d1c03c-734f-42e3-98ac-d7e50594a946/istio-proxy/0.log" Oct 30 18:33:18.617390 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:18.617344 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98bcg_320a0cd8-4ba1-4247-b0c7-3cfc8876b471/dns/0.log" Oct 30 18:33:18.740572 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:18.740537 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98bcg_320a0cd8-4ba1-4247-b0c7-3cfc8876b471/kube-rbac-proxy/0.log" Oct 30 18:33:19.125779 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:19.125703 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m72vs_26243dda-d8a0-457d-8844-81a32a6def00/dns-node-resolver/0.log" Oct 30 18:33:19.910568 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:19.910541 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-b7p7t_a7411194-34e7-4d6e-8415-efb9c6e32fc1/istio-proxy/0.log" Oct 30 18:33:20.606989 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:20.606957 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgdc6_d078773f-7b19-4a09-a258-a996048df5aa/node-ca/0.log" Oct 30 18:33:22.861406 ip-10-0-143-147 kubenswrapper[4622]: I1030 18:33:22.861373 4622 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-575956cc9cpm72l_43d1c03c-734f-42e3-98ac-d7e50594a946/istio-proxy/0.log"